Senior Data Engineer
Core
Design and implement scalable, well-governed data pipelines and reconciliation solutions for the insurance sector, transitioning legacy reporting capabilities to sustainable platforms.
Role type
Senior hands-on data engineer
Builds
Production-grade data pipelines, data mappings, lineage, and automated reconciliation processes
Domain
Insurance / Cloud Data Engineering
Required skills
PySpark, SQL, Databricks, GCP data technologies, dimensional/hierarchical data modelling, automated data-quality checks, data reconciliation, CI/CD, observability, entity matching
Preferred skills
Insurance domain knowledge, Microsoft Dynamics 365 integration, Power BI, Slowly Changing Dimensions (SCDs), master/reference data management
Technologies
GCP, Databricks, PySpark, SQL, Microsoft Dynamics 365, Power BI, Git, CI/CD
Responsibilities
Re-engineer ingestion pipelines from operational data; Design and implement robust data mappings and lineage; Develop automated data reconciliation processes; Refactor PySpark-based data processing with testing and versioning; Collaborate with SMEs to expand data coverage; Assess and implement GCP data pipeline options; Source mappings from Microsoft Dynamics 365; Support changes to reporting hierarchies; Build automated data-quality checks and monitoring; Produce technical documentation and handover materials.
Seniority
Senior, hands-on IC