Sr. Evaluation Engineer
Core
Design and build evaluation systems, pipelines, and frameworks to guide the development, testing, and release of Edwin AI, an AI-powered observability and incident intelligence platform.
Role type
Senior AI Evaluation Engineer
Builds
Production-grade evaluation pipelines, golden datasets, automated graders, and regression frameworks for AI agents and retrieval systems.
Domain
IT Infrastructure Management / AI Observability
Deliverable
production ML models
Required skills
Python engineering, AI evaluation frameworks (LangSmith, Arize, DeepEval, etc.), LLMs and agents, retrieval-augmented generation, prompt engineering, tool calling, regression testing, CI/CD integration, behavioral drift detection, LLM-as-a-judge techniques.
Preferred skills
Experience with multi-step/multi-agent AI systems, translating expert judgment into test cases, establishing evaluation-driven development practices.
Technologies
Python, LangSmith, Arize Phoenix, Braintrust, DeepEval, Ragas, TruLens, OpenAI Evals, MLflow.
Responsibilities
Define quality metrics for incident diagnostics and root-cause analysis; build offline and online evaluation pipelines; create and maintain golden datasets and regression suites; design step-level and trajectory-level evaluations for multi-agent workflows; monitor AI quality and behavioral drift in production; mentor other engineers on evaluation practices.
Seniority
Senior, hands-on IC
