Senior Software Engineer, Spark Platform
Core
Architecting and operating DoorDash's in-house Apache Spark ecosystem (runtime, shuffle service, scheduler) to power data, analytics, and ML workloads at scale.
Role type
Senior Staff Software Engineer (Distributed Systems/Platform)
Builds
High-scale distributed compute platform (Spark-on-Kubernetes) serving company-wide data and ML pipelines
Domain
Big Data Infrastructure / Distributed Systems
Deliverable
production ML models | infrastructure
Required skills
Apache Spark internals, distributed systems design, Kubernetes operators, multi-tenant scheduling, Scala/Java/Python/Go, SQL
Preferred skills
Remote shuffle systems (Celeborn, Magnet), batch schedulers (YuniKorn, Volcano), data lake formats (Iceberg, Delta Lake)
Technologies
Apache Spark, Kubernetes, Scala, Java, Python, Go, SQL, YuniKorn, Volcano, Iceberg, Delta Lake
Responsibilities
Set multi-year technical direction for the Spark platform, solve deep distributed-systems problems (shuffle, scheduling, failure modes), partner with Engineering Manager on roadmap and hiring, mentor the team, represent the platform in cross-team architecture forums
Seniority
Senior, hands-on IC with strategic leadership
