CareerPlan

Lead Data Engineer, Databricks

USRemoteFull-time2026-11-16
Remote · from United States

Core

Lead data engineering workstreams for enterprise customers, delivering production-grade scalable batch and streaming pipelines on Databricks.

Role type

Senior IC Lead Data Engineer (Databricks)

Builds

Scalable batch and streaming data pipelines, Medallion architecture data layers

Domain

Cloud data engineering, Databricks ecosystem

Deliverable

production ML models

Required skills

Python, SQL, Apache Spark, Delta Lake, Cloud platforms (AWS/Azure/GCP), Data modeling, CI/CD, Performance tuning, Technical leadership

Preferred skills

Healthcare/Life sciences domain, Unity Catalog, Lakeflow, Databricks Asset Bundles, dbt, Airflow, Legacy ETL migration

Technologies

Databricks, PySpark, SQL, Delta Lake, AWS, Azure, GCP, Git, CI/CD pipelines

Responsibilities

Lead data engineering activities from design to production handover; Build and optimize batch/streaming pipelines; Implement Medallion architecture; Optimize platform performance and cost; Implement data quality processes as code; Mentor engineers; Partner with Data Architects; Communicate technical decisions to stakeholders

Seniority

Senior, hands-on IC with leadership