Senior Data Engineer
Core
Design, develop, and optimize secure, scalable, and high-performance data solutions and end-to-end data pipelines for the Department of Planning, Housing and Infrastructure.
Role type
Senior Data Engineer
Builds
End-to-end data pipelines, data solutions within Microsoft Fabric (Lakehouse, Warehouse, Pipelines, Notebooks), and serverless integration solutions.
Domain
Public sector / Modern Data Platform
Deliverable
production ML models | product features | infrastructure
Required skills
Microsoft Fabric (Lakehouse/Warehouse/Pipelines/Notebooks), Advanced SQL (query optimization, indexing), Azure Data Factory (ADF), Python, PySpark, CI/CD pipelines (Azure DevOps), Data Security & Governance (RBAC, Managed Identities, Data Masking), Serverless development (Azure Functions, Logic Apps)
Preferred skills
Experience migrating existing Azure-based data platforms to Microsoft Fabric
Technologies
Microsoft Fabric, Azure Data Factory, Azure SQL, Azure Functions, Logic Apps, Azure DevOps, Python, PySpark, Delta Lake, OneLake
Responsibilities
Design and implement data solutions within Microsoft Fabric; migrate existing Azure-based data platforms to Fabric; develop and optimize ETL/ELT pipelines; enforce data security and compliance; collaborate with engineering, analytics, and architecture teams to translate requirements into scalable solutions.
Seniority
Senior, hands-on IC
