Lead Data Engineer - Data Engineering 4C
Core
Develop PySpark jobs for historical data extraction, transformation, and load from 29 source systems into Databricks bronze/silver layers.
Role type
Lead Data Engineer
Builds
Reusable PySpark frameworks for source-to-harmonized data migration
Domain
Data Engineering / Cloud Computing
Deliverable
production ML models | infrastructure
Required skills
PySpark, Databricks Platform, Snowflake (Platform), Data Engineering, Cloud Computing, Agile Methodology
Preferred skills
Data Literacy, Data Engineering best practices
Technologies
PySpark, Databricks, Snowflake, Azure
Responsibilities
Develop PySpark jobs for historical data extraction, transformation and load; Optimise large-scale batch ingestion into Databricks bronze/silver layers; Build reusable PySpark frameworks for source-to-harmonized data migration; Assist in writing PySpark scripts for historical data load and validation; Support data quality checks and reconciliation between source and Databricks layers
Seniority
Lead, hands-on IC with mentorship responsibilities