Evaluation Engineer
Core
Build and run automated tests to assess the quality, reliability, and performance of AI applications on legal and business tasks.
Role type
Mid-to-Senior IC Evaluation Engineer (AI/LLM)
Builds
Automated evaluation cases, test datasets, and regression suites for AI workflows
Domain
Legal technology + Large Language Models (LLM)
Deliverable
production ML models
Required skills
Python, Java, C#, Large Language Models (LLM) fundamentals, APIs, Git, CI/CD tools (Jira, Jenkins), Statistics (sampling, precision, recall), Analytical skills
Preferred skills
Coding agents, Prompt engineering, Automated grading calibration, Regression testing
Technologies
Python, Java, C#, Jira, Jenkins, Git
Responsibilities
Implement automated evaluation cases and scripts; Prepare and maintain test datasets with legal specialists; Run evaluations of answer accuracy, citations, and retrieval relevance; Execute functional and integration tests for AI workflows; Review outputs and execution traces to identify and reproduce failures; Compare results across model and application versions; Support calibration of automated graders; Maintain regression suites and CI/CD integration
Seniority
Mid-to-Senior, hands-on IC
