Senior Lead Data Engineer - Databricks
Core
Design and deliver modern, scalable data platforms and Lakehouse architectures for clients using Azure and Databricks.
Role type
Senior Lead Data Engineer (Databricks/Azure)
Builds
End-to-end data engineering solutions, batch and streaming pipelines, enterprise-grade Data Warehouses, and Lakehouses.
Domain
Cloud Data Engineering (Azure/Databricks)
Deliverable
production ML models | product features | dashboards & analysis
Required skills
Data Engineering, Lakehouse Architecture, PySpark, SQL, Azure Data Factory, Delta Lake, Medallion Architecture, Data Governance, CI/CD, Technical Mentoring
Preferred skills
Snowflake, Structured Streaming, Kafka, Databricks Workflows, Delta Live Tables, Unity Catalog, MLflow, dbt, Airflow
Technologies
Azure Databricks, Apache Spark, PySpark, SQL, Python, Delta Lake, Azure Data Factory, Azure Synapse Analytics, ADLS Gen2, Snowflake, Azure DevOps, Microsoft Purview, Azure Key Vault, Microsoft Entra ID, Kafka, Airflow, dbt, MLflow, Docker, Kubernetes, Power BI
Responsibilities
Lead design and development of scalable data engineering solutions; Design and implement modern Lakehouse architectures; Build and optimize large-scale batch and streaming data pipelines; Develop ETL/ELT pipelines using Databricks, PySpark, SQL, and Azure Data Factory; Design, implement, and optimize Delta Lake solutions; Collaborate with clients to translate business needs into technical solutions; Optimize Spark workloads for performance and cost; Implement data governance, security, and metadata management; Establish CI/CD pipelines and DevOps practices; Mentor and coach junior data engineers.
Seniority
Senior, hands-on IC with leadership responsibilities
