Senior AI Engineer, Quality
Core
Building the unified evaluation infrastructure, automated pipelines, and production feedback loops to ensure AI agents perform reliably at enterprise scale for audit and advisory workflows.
Role type
Senior IC AI Engineer (Evaluation & Quality)
Builds
Unified evaluation platform, automated evaluation pipelines, observability systems, and production feedback loops for AI agents.
Domain
Financial audit, cybersecurity, privacy, and AI/ML engineering.
Deliverable
production ML models
Required skills
TypeScript, React, Python, Postgres, LLM orchestration, agent evaluation frameworks, observability/tracing, prompt engineering, data-driven decision making.
Preferred skills
Experience shipping production LLM features, designing evaluation harnesses for complex agentic systems, building guardrails and monitoring systems.
Technologies
LangSmith, LangGraph, TypeScript, React, Python, Postgres.
Responsibilities
Design and build a unified evaluation platform as the single source of truth for agentic systems; build observability systems to surface agent behavior and failure modes; implement automated pipelines to evaluate new models against critical workflows; design comparison frameworks to measure effectiveness, consistency, latency, and cost; define and document evaluation standards and best practices for the engineering organization.
Seniority
Senior, hands-on IC