Lead Data Platform Engineer - AWS, Spark
Core
Lead a technical team building AWS-based data pipelines and data lake components for fraud-related data assets.
Role type
Lead Data Platform Engineer (Hands-on IC with team leadership)
Builds
AWS data pipelines, data lake components, producer-side data platform capability
Domain
Financial services / Data Engineering / Cloud Platform
Deliverable
production ML models
Required skills
AWS data services (Glue, S3, Step Functions, IAM, Lake Formation), Apache Spark, Scala/Python, Terraform, CI/CD, Infrastructure as Code, Cloud security principles
Preferred skills
AWS Developer Certification, Kubernetes/OpenShift, automated testing (pytest/Scalatest), code quality scanning (SonarQube/Veracode), RESTful APIs, data mesh concepts
Technologies
AWS, Spark, Scala, Python, Terraform, Jenkins, Git, Snowflake, OpenShift, Kubernetes
Responsibilities
Lead and support an agile technical team delivering data platform solutions; Guide delivery of AWS-based data pipelines and data lake components; Support handover and maintenance of cloud components; Lead building, testing, and maintaining high-quality data pipelines; Ensure accurate metadata preparation and publication; Champion DevSecOps practices, CI/CD, and Infrastructure as Code; Mentor and coach senior and junior engineers; Promote modern engineering practices including automation and AI-assisted development
Seniority
Senior, hands-on IC with team leadership
