Staff AI Engineer
Core
Build a real-time evaluation engine that judges AI agent quality and enforces safety guardrails using production traces.
Role type
Staff AI Engineer (LLM Observability & Evaluation)
Builds
Real-time agent evaluation platform, safety guardrails, and feedback datasets for model iteration.
Domain
Financial Services / LLM Observability / Real-time Data Infrastructure
Deliverable
production ML models | infrastructure
Required skills
SRE, real-time data infrastructure, distributed tracing, Open Telemetry, streaming data processing, LLM observability platforms, traffic routing, anomaly detection
Preferred skills
Regulated industry experience, AI security controls, AI governance, open-source contributions
Technologies
Open Telemetry, Langfuse, MLFlow
Responsibilities
Build ingestion paths for high-volume agent traces, score agent behavior live, enforce quality/safety guardrails in the request path, correlate spans across services, convert production traces into test cases, set technical direction for LLM evaluation
Seniority
Staff, hands-on IC with strategic direction