Senior Software Engineer- ML Network Stack, ML Network Stack - Annapurna Labs
Core
Senior engineer building and maintaining the infrastructure that monitors and reports on functionality and performance of massive testing workloads for EC2 distributed AI/ML systems.
Role type
Senior IC infrastructure engineer (ML/HPC systems)
Builds
Automated dashboards, alerting mechanisms, and performance monitoring tools for ML/HPC workloads
Domain
Cloud infrastructure, High-Performance Computing (HPC), AI/ML systems
Deliverable
dashboards & analysis
Required skills
Linux, networking, C/C++, Rust, Python, CI/CD automation, system architecture, root cause analysis, SW/HW co-design
Preferred skills
Embedded systems, high-speed networking, RDMA interconnects, automated visualization (Grafana)
Technologies
NCCL, NVSHMEM, NIXL, AWS Managed Grafana, Athena, Linux, C/C++, Rust, Python
Responsibilities
Automate software delivery to customers using internal CI/CD tools; write Python code to spool up large clusters and run benchmarks; digest performance data to create dashboards; invent automatic alerting mechanisms for regressions; manage complexity across many instance types and software stacks; root-cause fleet-wide latency anomalies; lead architecture design reviews for multi-year initiatives.
Seniority
Senior (SDE III), hands-on IC with mentorship responsibilities