Software Engineer, Spark Platform
Core
Build and operate DoorDash's in-house Apache Spark ecosystem, including runtime, scheduler, and reliability tooling at company-wide scale.
Role type
Senior IC distributed systems engineer (Spark Platform)
Builds
In-house Spark platform, multi-tenant scheduling, cluster lifecycle automation, and observability tooling
Domain
Big data infrastructure, distributed systems, cloud computing
Deliverable
infrastructure
Required skills
Apache Spark operations, Kubernetes, distributed systems, Python, Go, Scala, Java, SQL, observability stacks, cloud networking
Preferred skills
Batch or big-data schedulers (YuniKorn, Volcano, Kueue), Spark-on-Kubernetes operator, SLO/SLI definition
Technologies
Apache Spark, Kubernetes, Prometheus, OpenTelemetry, AWS
Responsibilities
Execute Spark runtime upgrades and performance optimization, drive multi-tenant scheduling and executor bin-packing, own cluster lifecycle automation, build observability and incident automation, partner on shuffle and architecture work
Seniority
Senior, hands-on IC
