Lead Data Engineer
Core
Design, build, and evolve secure, scalable cloud-based data platforms for UK Public Sector and Utilities clients, supporting operational, analytical, and AI-driven use cases.
Role type
Senior IC Lead Data Engineer
Builds
Cloud-native data platforms, batch and real-time streaming pipelines
Domain
Cloud Data Engineering, UK Public Sector, Utilities
Deliverable
production ML models
Required skills
Real-time streaming (Kafka, Kinesis, Flink, Spark Streaming), AWS services (EC2, RDS, Lambda, Athena, Glue, S3, IAM), Python, SQL, Data modeling, CI/CD, Infrastructure as Code, Agile, Regulated environment experience
Preferred skills
Terraform, Docker, Kubernetes, Ansible, Packer, Linux/Windows administration, AI/ML platform support
Technologies
Apache Kafka, AWS Kinesis, Apache Flink, Spark Streaming, Apache Pulsar, AWS EC2, AWS RDS, AWS Lambda, AWS Athena, AWS Glue, AWS S3, AWS IAM, Python, SQL, Git, Terraform, Docker, Kubernetes, Ansible, Packer
Responsibilities
Lead design and deployment of secure data pipelines; Design and implement batch and real-time streaming solutions; Own technical design decisions; Lead technical workshops and architecture discussions; Assess and modernize client data platforms; Build data ingestion and integration capabilities; Ensure data quality, governance, and security; Drive engineering best practices (code reviews, automated testing, CI/CD); Mentor engineers; Perform data modeling and performance tuning
Seniority
Senior, hands-on IC with technical leadership
