Lead Data Engineer
Core
Drive performance, governance, and AI-native maturity of an enterprise Data Platform on Databricks.
Role type
Senior IC Data Engineer (Databricks)
Builds
Scalable data pipelines, Lakehouse solutions, and AI-assisted operational automation on Databricks.
Domain
Data Engineering / AI Operations / Cloud Data Platforms
Deliverable
production ML models | infrastructure
Required skills
Databricks Lakehouse architecture, Delta Lake, Unity Catalog, Spark workload tuning, PySpark, SQL, CI/CD for data, observability tooling
Preferred skills
Databricks-native AI capabilities, agentic frameworks, Databricks Serverless Compute, Infrastructure-as-Code
Technologies
Databricks, Delta Lake, Unity Catalog, PySpark, SQL
Responsibilities
Design and develop scalable data pipelines and Lakehouse solutions; Tune Databricks workloads for performance and cost; Establish and enforce best practices for partitioning, clustering, and workload isolation; Track performance trends and resolve long-running loads; Design and operationalize Unity Catalog for data governance; Build monitoring and self-healing automation using Databricks-native AI; Drive CI/CD workflows for Databricks assets; Lead design reviews and mentor Data Engineers; Own Databricks vendor coordination.
Seniority
Senior, hands-on IC with mentorship responsibilities
