CareerPlan

Lead Data Engineer

Cochin, KakkanadFull-time2026-10-12 → 2026-11-16

Core

Lead the design and delivery of advanced data engineering projects for clients, focusing on optimizing large-volume pipelines, diagnosing performance issues, and ensuring SLA compliance.

Role type

Lead Data Engineer (Spark/PySpark)

Builds

Scalable data pipelines, reusable configuration-driven components, and platform infrastructure (object storage, containerization, CI/CD)

Domain

Financial services / Data Engineering

Deliverable

production ML models

Required skills

PySpark, Spark internals, Delta Lake, Iceberg, Python, distributed systems, time-series data, Kubernetes, CI/CD

Preferred skills

Trade surveillance, capital markets, KDB/q, MinIO, S3, regulatory compliance

Technologies

PySpark, Delta Lake, Iceberg, Kubernetes, MinIO, S3, CI/CD

Responsibilities

Lead design and delivery of client data engineering projects; Design pipelines meeting SLAs; Diagnose failures and performance problems; Set data layout and storage strategy; Build reusable components; Review team code and set standards; Work with client teams to define requirements; Contribute to platform work (storage, containerization, CI/CD); Mentor developers and document designs.

Seniority

Senior, hands-on IC with team leadership