Senior Principal Site Reliability Engineer (Chaos Engineering)
Core
Design and build an enterprise-grade chaos engineering platform from 0 to 1 to enable fault injection and resilience validation across 2,000+ microservices, aiming to reduce P0/P-1 severity incidents by over 50%.
Role type
Senior Principal Site Reliability Engineer (Chaos Engineering)
Builds
Enterprise-grade chaos engineering platform supporting multi-cluster (K8s + EC2 hybrid), multi-region, and multi-environment deployments
Domain
Cloud Infrastructure / Distributed Systems / Financial Technology
Required skills
Kubernetes fault injection (Chaos Mesh/Litmus), Go, distributed system failure modes, observability stack (Prometheus/Grafana/Thanos/OpenTelemetry), platform-level system architecture design
Preferred skills
Financial/trading system stability, SLO/Error Budget frameworks, automated fault recovery (self-healing), AWS infrastructure (EC2/EKS/Multi-AZ/Multi-Region), Netflix Chaos Engineering/AWS FIS/Gremlin, open-source contributions
Technologies
Kubernetes, Chaos Mesh, Litmus, Go, Prometheus, Grafana, Thanos, OpenTelemetry, EC2, EKS
Responsibilities
Design and build fault injection engine (Pod/Node/AZ level, network latency/packet loss/partition), implement production safety assurance (blast radius control, Kill Switch, automatic rollback), design experiment orchestration for complex fault scenarios, define safety standards and approval workflows for mainnet injection, execute routine chaos experiments (daily patrol, periodic validation, large-scale drills), mentor 2–3 engineers in chaos engineering
Seniority
Senior Principal, hands-on IC with strategy & mentorship