Lead Engineer, AI
Core
Build evaluation frameworks, observability tooling, and diagnostic infrastructure to measure, diagnose, and improve the quality of production AI agents and systems.
Role type
Lead Engineer (AI Quality Engineering)
Builds
Evaluation infrastructure, CI/CD pipelines for AI evals, diagnostic tooling, and datasets for AI agents.
Domain
AI/LLM engineering, observability, and quality assurance
Deliverable
production ML models
Required skills
Building production software with LLM-powered agents, designing evaluation/observability infrastructure for ML systems, designing datasets and annotation workflows, running structured experiments with prompts and models, leading and managing engineering teams, proficiency in Ruby on Rails and Python
Preferred skills
Experience with evaluation frameworks like Braintrust or LangSmith, experience with AI-generated or simulated conversations
Technologies
Ruby on Rails, Python, Braintrust, LangSmith
Responsibilities
Build and own evaluation infrastructure including CI/CD pipelines and scorers; Diagnose quality breakdowns across agent pipelines; Grow datasets via annotation workflows and AI-generated simulations; Run structured experiments to evaluate models and optimize cost/latency; Lead and grow the AI Quality engineering team; Partner with AI Core engineering to ensure quality improvements.
Seniority
Lead, hands-on player-coach with team management
