CareerPlan

Lead Data Engineer, Databricks (Remote, US)

United StatesRemoteFull-time2026-11-13 → 2026-11-16

Core

Lead the build of production Lakehouse pipelines on customer engagements, translating architecture into working pipelines and owning the quality of what ships.

Role type

Lead Data Engineer (Databricks)

Builds

Production Lakehouse pipelines (batch and streaming) on customer engagements

Domain

Data Engineering / Cloud Data Platforms

Deliverable

production ML models

Required skills

Python, SQL, Apache Spark, Delta Lake, AWS/Azure/GCP, Data Modeling, Git, CI/CD, Performance Optimization, Cost Management

Preferred skills

Healthcare/Life Sciences experience, Databricks Certifications, Unity Catalog, Lakeflow, Databricks Asset Bundles, dbt, Airflow, Legacy ETL Migration, Consulting background

Technologies

Databricks, PySpark, Delta Lake, AWS, Azure, GCP, Lakeflow, Unity Catalog, dbt, Airflow

Responsibilities

Lead engineering on customer engagements from design through production handover; Build and optimize batch and streaming pipelines; Implement Medallion architecture; Own performance and cost optimization; Implement data quality and validation as code; Mentor mid-level and junior engineers; Partner with Data Architect on design decisions

Seniority

Senior, hands-on IC with leadership responsibilities