Staff Software Engineer – Testing & Evals Frameworks
Core
Design and own the architecture and strategy for evaluation frameworks, automation tools, and testing infrastructure that enable safe, rapid shipping across functional, end-to-end, LLM/eval, and security-focused testing practices.
Role type
Staff Software Engineer (Testing & Evals Frameworks)
Builds
Evaluation frameworks, automation tools, testing infrastructure, and CI/CD quality gates
Domain
Identity verification, distributed systems, AI/LLM evaluation
Deliverable
production ML models
Required skills
Python, TypeScript, JavaScript, Java, Playwright, Jest, API automation, CI/CD integration, distributed systems, event-driven architectures, OAuth 2.0, OIDC, RBAC/ReBAC, secrets management, canary deployments, synthetic monitoring, LLM eval frameworks, agentic tools, RAG pipelines
Preferred skills
Temporal, GCP, Docker, Kubernetes, observability, chaos engineering, contract testing, performance testing, AI agent red-teaming, fraud detection
Technologies
Playwright, Jest, Python, TypeScript, Java, GitHub Actions, ArgoCD, Temporal, OAuth 2.0, OIDC, SAML 2.0, SCIM 2.0, Okta, Entra ID, Auth0, Ping, Zanzibar, SpiceDB, Datadog, OpenTelemetry, LangChain, LangGraph, CrewAI, BrainTrust, Claude API, Anthropic SDK, Locust, k6, Gatling, Spring Boot, Ruby on Rails
Responsibilities
Set architecture and long-term strategy for automation frameworks and testing infrastructure; Define end-to-end and integration testing strategy for complex, asynchronous, event-driven systems; Build and champion canary deployments, progressive/phased rollouts, synthetic monitoring, and automated rollback triggers; Set the standard for testing authentication/authorization flows, RBAC/ReBAC, secrets management, and tenant isolation; Integrate automated testing into CI/CD pipelines org-wide; Build robust evaluation pipelines for AI-assisted features and LLM agents; Partner with Security, Platform, and Product teams as a testing strategist; Guide senior and mid-level engineers on test architecture, testing strategy, and quality practices; Debug and root-cause issues across frontend, backend, APIs, and distributed services; Define and track reliability, quality, and eval accuracy metrics
Seniority
Staff, hands-on IC with mentorship
