Senior Systems Software Engineer - Accelerated Spark Applications
Core
Design, build, and operate a hosted Apache Spark service control plane on Kubernetes to accelerate data processing on GPUs for NVIDIA's internal data platform.
Role type
Senior Systems Software Engineer (Distributed Systems & GPU Acceleration)
Builds
API-based Spark service, cuDF Spark plugin, CPU/GPU workload router, and deployment infrastructure on Kubernetes.
Domain
Cloud computing, Big Data, GPU acceleration, Distributed Systems
Deliverable
production ML models
Required skills
Java, Scala, C++, Python, Kubernetes, Distributed Systems, REST APIs, Spark Connect, Kyuubi, Delta Lake, Apache Iceberg, CUDA
Preferred skills
Apache Spark internals, Unity Catalog, Velox, Presto, Apache Arrow, Apache DataFusion, AI tools in development
Responsibilities
Design and build the Spark service control plane; own code reviews for end-to-end service reliability; extend the cuDF runtime and workload router; investigate and fix production failures on large-scale clusters; maintain metrics, dashboards, and on-call runbooks; provide feedback on CI/CD and testing strategies.
Seniority
Senior, hands-on IC