Lead Data Engineer
Core
Design, build, and optimize scalable data pipelines and platforms while leading a small offshore data engineering team.
Role type
Senior hands-on IC data engineer with team leadership
Builds
Scalable lakehouse architectures, high-performance data models, batch and real-time data pipelines
Domain
Cloud data platforms (Azure), distributed data processing
Deliverable
production ML models | product features | infrastructure
Required skills
Databricks, Spark, Python, Scala, SQL, performance tuning, data modeling, streaming architectures, data quality frameworks, CI/CD
Preferred skills
Kafka/Confluent, Snowflake, AI/ML pipelines, feature engineering, regulated industry experience
Technologies
Databricks, Spark, Azure, Kafka, Confluent, Snowflake
Responsibilities
Design and optimize complex batch and real-time data pipelines; Build and maintain scalable lakehouse architectures; Provide technical leadership and coordinate offshore teams; Define tasks and review deliverables for engineering standards; Implement advanced ETL/ELT frameworks and reusable patterns; Solve complex data challenges including performance tuning and partitioning; Implement data quality, validation, and observability frameworks; Collaborate with API, Analytics, and Platform teams; Drive best practices in coding standards, testing, and CI/CD; Troubleshoot production issues; Act as liaison between onshore stakeholders and offshore teams
