Lead Data Engineer, Databricks (Remote, US)
Core
Lead the build of production Lakehouse pipelines on customer engagements, translating architecture into working pipelines and owning the quality of what ships.
Role type
Lead Data Engineer (Databricks)
Builds
Production Lakehouse pipelines (batch and streaming) on customer engagements
Domain
Data Engineering / Cloud Data Platforms
Deliverable
production ML models
Required skills
Python, SQL, Apache Spark, Delta Lake, AWS/Azure/GCP, Data Modeling, Git, CI/CD, Performance Optimization, Cost Management
Preferred skills
Healthcare/Life Sciences experience, Databricks Certifications, Unity Catalog, Lakeflow, Databricks Asset Bundles, dbt, Airflow, Legacy ETL Migration, Consulting background
Technologies
Databricks, PySpark, Delta Lake, AWS, Azure, GCP, Lakeflow, Unity Catalog, dbt, Airflow
Responsibilities
Lead engineering on customer engagements from design through production handover; Build and optimize batch and streaming pipelines; Implement Medallion architecture; Own performance and cost optimization; Implement data quality and validation as code; Mentor mid-level and junior engineers; Partner with Data Architect on design decisions
Seniority
Senior, hands-on IC with leadership responsibilities
