Building robust large-scale social media crawler for various social media platforms
Maintain day-to-day operation of said crawlers alongside our data quality team to make sure the completeness and timeliness of incoming data
Work in team along with several stakeholders to better understand and communicate about data requirements and issues
Qualifications
Intermediate or advanced understanding of SQL
Have proficiency in several programming languages, mainly in Javascript (Node) and Python. Having a good experience in using Scala for production (not related to Spark) is a big plus
Have intermediate or advanced understanding of several data-related technologies such as MariaDB, Postgres, Redis, Kafka, and Elasticsearch
Have ample experience using Linux-based system
Have understanding on several data acquisition techniques such as webscraping, API-based data consumption, and so on
