Senior Software Engineer, Spark Platform
Core
Set multi-year technical direction for DoorDash's in-house Spark-on-Kubernetes platform, solving distributed-systems problems involving runtime performance, shuffle architecture, multi-tenant scheduling, and large-scale failure modes.
Role type
Senior IC platform engineer (distributed systems)
Builds
In-house Spark-on-Kubernetes platform serving data, analytics, and ML workloads
Domain
Big Data / Distributed Systems
Deliverable
production ML models
Required skills
Apache Spark internals, Kubernetes operator patterns, distributed system architecture, Scala/Java/Python/Go, SQL, platform operations, multi-tenant scheduling, remote shuffle systems, data lake table formats (Iceberg/Delta Lake)
Preferred skills
SRE patterns, observability for distributed compute, technical leadership experience
Technologies
Apache Spark, Kubernetes, Amazon EMR, Databricks, Apache Iceberg, Delta Lake
Responsibilities
Define technical roadmap and architecture for the Spark platform, mentor engineers through design reviews, partner with Engineering Manager on hiring and team shape, represent platform in cross-team architecture forums, solve large-scale distributed-systems problems
Seniority
Senior, hands-on IC with technical leadership
