Sr Data Engineer
Core
Design, develop, and maintain production data pipelines on Databricks, building bronze/silver/gold layers, data products, and ML feature tables.
Role type
Senior IC data engineer (lakehouse)
Builds
Production data pipelines, curated datasets, dimensional models, and ML training/feature tables
Domain
Cloud data platform / Data engineering
Deliverable
production ML models
Required skills
Python, SQL, Databricks, Delta Lake, dbt, Dagster, dimensional modeling, data quality, CI/CD, orchestration
Preferred skills
Databricks certification, Unity Catalog, declarative Python ingestion frameworks (dlt, Airbyte)
Technologies
Databricks, Python, SQL, Apache Spark, Delta Lake, dbt, Dagster, Azure DevOps, Azure Key Vault
Responsibilities
Design and maintain production data pipelines on Databricks; Build Bronze-layer ingestion from APIs, databases, and cloud storage; Develop Silver-layer transformations in dbt and Python; Create Gold-layer data products and ML feature tables; Author and maintain data contracts using ODCS; Implement data quality as code; Orchestrate ingestion and transformation in Dagster; Apply governance through Unity Catalog; Tune Spark jobs and Delta Lake performance; Troubleshoot production failures and manage on-call rotation; Build and maintain CI/CD for data assets; Instrument pipelines for observability; Work with stakeholders to translate requirements into data solutions
Seniority
Senior, hands-on IC