Data Engineer
Core
Design, build, and maintain scalable data pipelines and governance frameworks to support regulatory processes and data-driven insights.
Role type
Senior Data Engineer (Cloud Data Platform)
Builds
Scalable data pipelines, metadata-driven ingestion frameworks, and governed data assets on Databricks.
Domain
Government / Regulatory / Cloud Data Engineering
Deliverable
production ML models
Required skills
Python, PySpark, Spark SQL, Databricks, Delta Lake, Unity Catalog, CI/CD, Data Modeling, Azure ADLS Gen2, Key Vault
Preferred skills
Lakeflow (Delta Live Tables), MLflow, Data Visualization (PowerBI, Tableau), Databricks/Azure Certifications
Technologies
Databricks, Python, PySpark, Spark SQL, Delta Lake, Unity Catalog, Azure ADLS Gen2, Key Vault, Lakeflow, MLflow, PowerBI, Tableau
Responsibilities
Design and build scalable data pipelines using Databricks and PySpark; Optimize cluster performance and cost; Implement data security and governance with Unity Catalog; Manage CI/CD workflows and incident resolution; Develop reusable components for data ingestion frameworks.
Seniority
Senior, hands-on IC