Lead Data Engineer
Core
Lead the design and delivery of advanced data engineering projects for clients, focusing on optimizing large-volume pipelines, diagnosing performance issues, and ensuring SLA compliance.
Role type
Lead Data Engineer (Spark/PySpark)
Builds
Scalable data pipelines, reusable configuration-driven components, and platform infrastructure (object storage, containerization, CI/CD)
Domain
Financial services / Data Engineering
Deliverable
production ML models
Required skills
PySpark, Spark internals, Delta Lake, Iceberg, Python, distributed systems, time-series data, Kubernetes, CI/CD
Preferred skills
Trade surveillance, capital markets, KDB/q, MinIO, S3, regulatory compliance
Technologies
PySpark, Delta Lake, Iceberg, Kubernetes, MinIO, S3, CI/CD
Responsibilities
Lead design and delivery of client data engineering projects; Design pipelines meeting SLAs; Diagnose failures and performance problems; Set data layout and storage strategy; Build reusable components; Review team code and set standards; Work with client teams to define requirements; Contribute to platform work (storage, containerization, CI/CD); Mentor developers and document designs.
Seniority
Senior, hands-on IC with team leadership