Data Engineer
Core
Design, build, and optimise reliable, performant, and secure data platform and pipelines enabling analytics and data-driven decision-making across the business.
Role type
Senior Data Engineer (Cloud & Lakehouse)
Builds
Production-grade data pipelines, datasets, and data products using Databricks and AWS.
Domain
Water utility / Public sector / Cloud Data Engineering
Deliverable
production ML models | product features
Required skills
Data ingestion and transformation pipelines, Analytical data models, Data quality checks and validation logic, Cloud-based data platforms (Databricks, AWS), Python, SQL, Agile/DevOps practices, Git-based version control, CI/CD pipelines, Automated testing.
Preferred skills
Apache Spark, Delta Lake, AWS certifications, Databricks certifications, Orchestration tools (Airflow, Lakeflow Jobs), Structure Streaming.
Technologies
Databricks, AWS (S3, Glue, Lambda, RDS, DynamoDB, API Gateway), Apache Spark, Delta Lake, Python, SQL, Git, Airflow.
Responsibilities
Design, build, and optimise data pipelines, datasets, and data products; Develop ELT pipelines using Spark, Delta Lake, Python, and SQL; Implement data modelling, data quality checks, validation, and observability; Deploy and support data pipelines across Dev, Test, and Prod environments using CI/CD; Execute testing and validation to ensure accuracy, reliability, and performance; Monitor and support production data products, conducting incident investigation and remediation; Participate in code reviews and apply software engineering standards.
Seniority
Senior, hands-on IC