Software Engineer, Performance Instrumentation
Core
Define and maintain performance data requirements, identify visibility gaps, and build low-overhead instrumentation and annotation APIs for profiling libraries, runtimes, and infrastructure telemetry.
Role type
Senior IC software engineer (performance instrumentation)
Builds
Low-overhead instrumentation and annotation APIs for profiling and telemetry
Domain
Accelerated computing and AI systems
Deliverable
production ML models
Required skills
CUDA, Python, C++, machine learning frameworks (PyTorch, JAX), annotation/tracing APIs (NVTX), data pipeline validation, overhead measurement, debugging across component boundaries
Preferred skills
Telemetry conventions/schemas, storage/checkpointing performance, low-overhead event collection, network/collective-communication telemetry (NCCL, NICs, fabric counters)
Technologies
CUDA, PyTorch, JAX, NVTX, NCCL
Responsibilities
Define and maintain performance data requirements from various sources; Identify and prove visibility gaps with concrete evidence; Inform the build of low-overhead instrumentation and annotation APIs; Ensure data quality at ingestion by implementing validation, sanitization, checks, and alerting; Measure and manage the overhead and observer effects of instrumentation on real workloads; Collaborate with multiple partner teams simultaneously
Seniority
Senior, hands-on IC
