Databricks Data Engineer Consultant
Core
Design, develop, and optimize scalable data ingestion pipelines and transformation workflows on Azure Databricks to enable advanced analytics and data engineering initiatives.
Role type
Senior IC Databricks Data Engineer
Builds
Scalable data pipelines, transformation workflows, and clean structured datasets for Data Science use cases
Domain
Financial services / Cloud Data Engineering
Deliverable
production ML models | product features
Required skills
Azure Databricks, Apache Spark (PySpark), Azure Data Factory, Delta Lake, Python, SQL, ETL/ELT processes, data modeling, CI/CD (Azure DevOps, Terraform)
Preferred skills
Scala, NoSQL, data lake architectures, Databricks certification, banking domain experience
Technologies
Azure Databricks, Azure Data Factory, Azure Synapse, Azure Blob Storage, Apache Spark, PySpark, Delta Lake, Azure DevOps, Git, Terraform
Responsibilities
Design and optimize data ingestion pipelines; Build and maintain Databricks notebooks for data preparation; Implement data quality checks and monitoring; Integrate data from structured, semi-structured, and unstructured sources; Optimize Spark jobs for performance and cost-efficiency; Document data pipelines and technical decisions
Seniority
Senior, hands-on IC