ML Engineer - Automated Evaluation and Adversarial Design
Core
Building and scaling automated evaluation systems and adversarial/stress-testing methodologies for AI-powered productivity and creative applications.
Role type
Senior IC machine-learning engineer (evaluation & adversarial design)
Builds
Automated evaluation frameworks, adversarial test suites, multi-turn stress-test pipelines, and quality assessment reports for AI features.
Domain
AI/ML evaluation, conversational AI, agentic workflows, productivity software
Deliverable
production ML models
Required skills
ML evaluation system design, adversarial/red-teaming methodology, multi-turn conversation analysis, stress testing, Python, PyTorch/TensorFlow, evaluation architecture, automation of evaluation pipelines
Preferred skills
Consumer AI feature evaluation, user workflow quality assessment, automated vs human evaluation alignment, scalable evaluation system design, API/custom model evaluation, agent orchestration frameworks (LangChain, LangGraph), observability tooling, tool-use reliability testing
Technologies
Python, PyTorch, TensorFlow, LangChain, LangGraph, CrewAI, AutoGen, LangSmith, Braintrust, Arize
Responsibilities
Define automated evaluation approaches for single-turn and multi-turn agentic experiences; Build adversarial test suites targeting model failure modes; Develop stress test protocols for atypical input conditions; Ensure alignment between automated and human evaluation methods; Collaborate with engineering to integrate evaluation into release workflows; Scale adversarial test case generation and execution; Communicate evaluation findings to cross-functional partners
Seniority
Senior, hands-on IC