Senior Data Engineer
Core
Design and develop batch and streaming ETL/ELT pipelines in Azure Databricks to support ingestion, transformation, curation, and delivery of manufacturing and enterprise data, while operationalizing ML workflows for data preparation, feature engineering, and model deployment.
Role type
Senior Data Engineer (ML Ops focus)
Builds
Bronze, Silver, and Gold data pipelines, data quality frameworks, governed data products in Unity Catalog, and production ML workflows.
Domain
Manufacturing / Enterprise Data / Cloud Data Engineering
Deliverable
production ML models | product features | infrastructure
Required skills
Scala, Python, SQL, Spark, Databricks, Unity Catalog, MLflow, CI/CD pipelines, Git-based workflows, batch and streaming ETL/ELT, data orchestration
Preferred skills
DLT, structured streaming, Event Hub, Kafka, Java, model governance, lineage, reproducibility, monitoring, validation checks, resilient pipeline design, manufacturing analytics, industrial IoT, plant-floor data
Technologies
Azure Databricks, Unity Catalog, MLflow, Git, Kafka, Event Hub, DLT
Responsibilities
Design and develop batch and streaming ETL/ELT pipelines; Build reusable Bronze, Silver, and Gold data pipelines; Build and operationalize ML workflows for data preparation and model deployment; Establish and maintain MLflow-based model lifecycle controls; Implement CI/CD pipelines for data products and ML code; Build and support Databricks jobs and workflows; Improve reliability, observability, and production readiness through testing and monitoring; Partner with data scientists and stakeholders to productionize scalable solutions; Support modernization of legacy data and ML workflows.
Seniority
Senior, hands-on IC