Senior Data Engineer - AWS & Databricks (Public Sector)
Core
Design, build, and operate dependable data pipelines and reusable data products on AWS and Databricks to support analytics, reporting, and machine learning.
Role type
Senior hands-on IC data engineer
Builds
Production-grade data pipelines, lakehouse architectures, and curated data products
Domain
Public sector data engineering on AWS and Databricks
Deliverable
production ML models
Required skills
Python, PySpark, SQL, Apache Spark, Delta Lake, AWS services (S3, Glue, IAM, Lambda, Step Functions), data modeling, incremental processing, data quality validation, workflow orchestration, cloud security
Preferred skills
Unity Catalog, Lakeflow, Delta Live Tables, Apache Airflow, dbt, Kafka, MLOps, data mesh concepts
Technologies
AWS, Databricks, Delta Lake, Unity Catalog, Terraform, Git, CI/CD
Responsibilities
Lead design and implementation of batch and streaming pipelines; Build lakehouse architectures using Delta Lake; Develop reusable data models and data products; Implement source integration and incremental processing; Establish data contracts and automated quality checks; Build and maintain orchestration workflows; Implement metadata, lineage, and access controls; Develop monitoring and operational runbooks; Optimise Spark and SQL workloads; Apply cloud security practices; Review designs and mentor engineers; Collaborate with stakeholders to define requirements.
Seniority
Senior, hands-on IC
