Lead Data Engineer, Databricks
Core
Lead data engineering workstreams for enterprise customers, delivering production-grade scalable batch and streaming pipelines on Databricks.
Role type
Senior IC Lead Data Engineer (Databricks)
Builds
Scalable batch and streaming data pipelines, Medallion architecture data layers
Domain
Cloud data engineering, Databricks ecosystem
Deliverable
production ML models
Required skills
Python, SQL, Apache Spark, Delta Lake, Cloud platforms (AWS/Azure/GCP), Data modeling, CI/CD, Performance tuning, Technical leadership
Preferred skills
Healthcare/Life sciences domain, Unity Catalog, Lakeflow, Databricks Asset Bundles, dbt, Airflow, Legacy ETL migration
Technologies
Databricks, PySpark, SQL, Delta Lake, AWS, Azure, GCP, Git, CI/CD pipelines
Responsibilities
Lead data engineering activities from design to production handover; Build and optimize batch/streaming pipelines; Implement Medallion architecture; Optimize platform performance and cost; Implement data quality processes as code; Mentor engineers; Partner with Data Architects; Communicate technical decisions to stakeholders
Seniority
Senior, hands-on IC with leadership