CareerPlan

Senior Principal Site Reliability Engineer (Chaos Engineering)

Kuala Lumpur Tech OfficeFull-time2026-11-16 → 2026-11-19

Core

Design and build an enterprise-grade chaos engineering platform from 0 to 1 to enable fault injection and resilience validation across 2,000+ microservices, aiming to reduce P0/P-1 severity incidents by over 50%.

Role type

Senior Principal Site Reliability Engineer (Chaos Engineering)

Builds

Enterprise-grade chaos engineering platform supporting multi-cluster (K8s + EC2 hybrid), multi-region, and multi-environment deployments

Domain

Cloud Infrastructure / Distributed Systems / Financial Technology

Required skills

Kubernetes fault injection (Chaos Mesh/Litmus), Go, distributed system failure modes, observability stack (Prometheus/Grafana/Thanos/OpenTelemetry), platform-level system architecture design

Preferred skills

Financial/trading system stability, SLO/Error Budget frameworks, automated fault recovery (self-healing), AWS infrastructure (EC2/EKS/Multi-AZ/Multi-Region), Netflix Chaos Engineering/AWS FIS/Gremlin, open-source contributions

Technologies

Kubernetes, Chaos Mesh, Litmus, Go, Prometheus, Grafana, Thanos, OpenTelemetry, EC2, EKS

Responsibilities

Design and build fault injection engine (Pod/Node/AZ level, network latency/packet loss/partition), implement production safety assurance (blast radius control, Kill Switch, automatic rollback), design experiment orchestration for complex fault scenarios, define safety standards and approval workflows for mainnet injection, execute routine chaos experiments (daily patrol, periodic validation, large-scale drills), mentor 2–3 engineers in chaos engineering

Seniority

Senior Principal, hands-on IC with strategy & mentorship