Lead Data Engineer
Core
Design, build, and operate modern data solutions on Databricks to power AI, Machine Learning, analytics, and enterprise insights.
Role type
Lead Data Engineer (Technical Lead + Hands-on IC)
Builds
Scalable batch and streaming data pipelines, governed data assets, and reusable engineering frameworks for AI/ML applications.
Domain
Data Engineering / Cloud Data Platforms
Deliverable
production ML models
Required skills
Databricks, Apache Spark/PySpark, Delta Lake, Python, SQL, Infrastructure as Code (Terraform), Data Governance, Workflow Orchestration, Data Modeling, CI/CD
Preferred skills
Cloud platforms (Azure/AWS/GCP), NoSQL, REST APIs, AI-assisted development tools, Insurance/Health data domain
Technologies
Databricks, Delta Lake, Unity Catalog, PySpark, Terraform, Git, Azure, AWS, GCP, Snowflake, Apache Iceberg, Apache Hudi, BigQuery
Responsibilities
Lead design and development of scalable data solutions; define engineering standards and patterns; mentor engineers; optimize production Spark workloads; implement data quality and governance; drive architectural alignment and technical decision-making.
Seniority
Senior, hands-on IC with technical leadership