Software Engineer, Spark Platform
Core
Build and operate an in-house Apache Spark platform at company-wide scale, managing runtime, multi-tenant scheduling, cluster lifecycle, and observability.
Role type
Senior Software Engineer (Distributed Systems/Big Data Platform)
Builds
In-house Spark ecosystem (runtime, scheduler, reliability tooling) serving data, analytics, and ML workloads.
Domain
Big Data / Distributed Systems / Cloud Infrastructure
Deliverable
production ML models
Required skills
Apache Spark operations, Kubernetes (operators, controllers), distributed system operations, Python/Go/Scala/Java, SQL, observability stacks (Prometheus, OpenTelemetry), cloud environment (AWS)
Preferred skills
Batch or big-data schedulers (YuniKorn, Volcano, Kueue), Spark-on-Kubernetes operator, SLO/SLI definition
Technologies
Apache Spark, Kubernetes, AWS, Prometheus, OpenTelemetry, YuniKorn, Volcano, Kueue
Responsibilities
Execute Spark runtime upgrades and performance optimization; Drive multi-tenant scheduling and executor bin-packing; Own cluster lifecycle automation (provisioning, upgrades, capacity, failure handling); Build observability and incident automation; Partner on shuffle, runtime, and architecture work.
Seniority
Mid-Senior, hands-on IC
