Senior Data Engineer ( Data Collection)
Core
Design, code, and manage end-to-end data ingestion pipelines and high-scale infrastructure to reliably ingest billions of events daily for Similarweb's digital intelligence platform.
Role type
Senior IC data engineer (data collection/ingestion)
Builds
High-scale data ingestion services and pipelines processing billions of events
Domain
Internet analytics / Big data infrastructure
Deliverable
production ML models | infrastructure
Required skills
Big data infrastructure development, high-scale system architecture, cloud provider expertise (AWS), pipeline design and management, system monitoring and health management, code review and technical mentorship
Preferred skills
Proficiency in Databricks, Spark, Airflow, Firehose, SQS, ability to learn new technologies quickly
Technologies
Databricks, Spark, Airflow, Firehose, SQS, AWS
Responsibilities
Design, code, and manage end-to-end data ingestion pipelines; develop and maintain modern data infrastructure; solve challenges of availability, reliability, and scalability; monitor production environment and ensure data quality; lead design and decision-making processes; mentor team members
Seniority
Senior, hands-on IC
