Senior Data Engineer
Core
Build and maintain scalable ETL pipelines, data models, and integration patterns on Azure Databricks to support the UK Public Sector's evolving data ecosystem.
Role type
Senior hands-on Data Engineer
Builds
Scalable data flows, ingestion pipelines, and curated data layers
Domain
Public Sector, Cloud Data Engineering (Azure)
Required skills
Python, PySpark, SQL, Azure Databricks, Azure Data Factory, Microsoft Purview, Event-driven ingestion, SOLID principles, Async programming, Unit/Integration testing, Secure ETL design, Power BI modeling
Preferred skills
UK Public Sector experience
Technologies
Azure, Databricks, PySpark, SQL, Azure Data Factory, Microsoft Purview, Event Grid, Power BI
Responsibilities
Build and maintain ETL pipelines and code in cloud data platforms; Support ingestion activity and onboarding of new data sources; Construct raw, refined and curated data layers; Validate solutions against functional and non-functional requirements; Deliver datasets, transformations and performance-optimised data products; Improve processes, engineering patterns, and reusable tooling; Monitor and measure pipeline performance; Ensure documentation meets acceptance standards
Seniority
Senior, hands-on IC