Senior AI Evaluation Engineer (CICD & Platform)
Core
Build continuous evaluation architecture, deployment pipeline infrastructure, and release gating for an enterprise AI Skills platform to ensure production AI skills execute accurately and safely.
Role type
Senior IC MLOps/Platform Engineer (AI Evaluation)
Builds
Automated testing harnesses, release-gating pipelines, and evaluation tooling for AI agents and skills.
Domain
Generative AI / MLOps / Enterprise Software
Deliverable
production ML models
Required skills
Python, CI/CD pipeline engineering, AI evaluation methodology, test automation frameworks, schema validation, synthetic data generation, LLM-as-a-Judge evaluation, Model Context Protocol (MCP), API contract testing, LLM security and governance
Preferred skills
LangSmith, Braintrust, DeepEval, Promptfoo, Phoenix, Pydantic, JSON Schema, OpenAPI, red-teaming utilities, production observability
Technologies
GitHub Actions, GitLab CI, pytest, LangSmith, Braintrust, DeepEval, Promptfoo, Phoenix, Pydantic, JSON Schema, OpenAPI
Responsibilities
Build tiered evaluation pipelines (static checks, synthetic multi-turn trajectories, LLM-as-a-Judge); Architect and maintain CI/CD build pipelines and runner environments; Develop synthetic data generation pipelines and automated red-teaming utilities; Curate versioned golden datasets and execute continuous regression testing; Ingest real-world production traces and failure logs into benchmark datasets.
Seniority
Senior, hands-on IC