Big Data Engineer
Core
Design, build, and own end-to-end data platform components including pipelines, tables, quality checks, and monitoring for analytical use cases.
Role type
Senior Big Data Engineer
Builds
Production data pipelines, dimensional data models, and reusable data components
Domain
Data Engineering / Cloud Infrastructure
Deliverable
production ML models | product features
Required skills
SQL (window functions, complex joins, execution plan analysis), Python (PySpark, pandas, boto3), Apache Spark on EMR, Apache Airflow, Data ingestion/CDC (DataX, Sqoop, Debezium), OLAP engines (StarRocks, Doris, ClickHouse), Data warehouse modeling (dimensional modeling, SCDs), AWS (S3, IAM, EMR), Linux, Git
Preferred skills
AWS cost optimization, Lakehouse formats (Iceberg, Hudi, Delta), Streaming (Kafka, Flink), dbt, Data lineage, QuickSight
Technologies
EMR, Airflow, DataX, StarRocks, Doris, ClickHouse, AWS S3, PySpark, Kafka, Flink, dbt, QuickSight
Responsibilities
Design and build ETL pipelines on EMR and Airflow; Model data using dimensional layering (ODS/DWD/DWS/ADS); Own data quality checks and freshness monitoring; Tune runtime performance and infrastructure costs; Build reusable components and automate repetitive work; Collaborate with analysts to translate requirements into usable datasets
Seniority
Senior, hands-on IC
