Senior AI Observability & Telemetry Engineer
Core
Architect and scale high-cardinality telemetry infrastructure for AI cloud and HPC datacenters, integrating hardware exporters and eBPF tools to monitor GPU clusters and network performance.
Role type
Senior IC observability engineer (AI/HPC infrastructure)
Builds
High-availability time-series databases, hardware-level exporters, eBPF diagnostic tools, automated alerting pipelines, and multi-tenant billing metric pipelines.
Domain
AI cloud computing, High-Performance Computing (HPC), Bitcoin mining infrastructure
Deliverable
production ML models
Required skills
Go, Kubernetes, Prometheus/OpenTelemetry, eBPF, Linux kernel tuning, time-series databases (VictoriaMetrics/Thanos/Mimir), GPU hardware metrics, network telemetry
Preferred skills
Site reliability engineering, technical leadership, cross-functional collaboration, high-velocity environment experience
Technologies
VictoriaMetrics, Thanos, Mimir, NVIDIA DCGM, IPMI, Redfish, Kubernetes, eBPF, BCC, Go
Responsibilities
Architect scalable telemetry infrastructure, integrate hardware exporters into observability stacks, build eBPF diagnostic tools, develop automated alerting pipelines, design metric pipelines for billing, lead technical design reviews and mentor team members
Seniority
Senior, hands-on IC with leadership responsibilities