Senior Data Engineer
Core
Design and deliver scalable Lakehouse architectures and production-grade data pipelines for defence, government, and national security clients using Databricks and cloud-native technologies.
Role type
Senior hands-on IC data engineer with technical leadership
Builds
Scalable Lakehouse architectures, production data pipelines, and enterprise data platforms
Domain
Defence, government, and national security data engineering
Required skills
Databricks, PySpark, Apache Spark, Delta Lake, Lakehouse architecture (Medallion), ETL/ELT, Azure/AWS, data governance, Unity Catalog, data modelling, CI/CD
Preferred skills
Technical leadership, mentoring, stakeholder engagement, emerging technology adoption
Technologies
Databricks, Apache Spark, PySpark, Delta Lake, Unity Catalog, Azure, AWS
Responsibilities
Design and build scalable data pipelines using Databricks and Spark; Develop modern Lakehouse architectures with Medallion design principles; Build robust ETL/ELT processes for large datasets; Drive data quality, monitoring, and operational support; Optimise platform performance, scalability, and cost; Integrate data platforms with cloud services and enterprise systems; Implement data governance, security, and access controls; Lead technical delivery across data engineering workstreams; Collaborate with data scientists and stakeholders to deliver end-to-end solutions; Support DevOps and CI/CD practices.
Seniority
Senior, hands-on IC with leadership responsibilities