Director, AI Automation Engineering
Core
Own the quality and reliability layer for AI agents and tools, building automated testing infrastructure, evaluation frameworks, and CI/CD guardrails to enable safe production releases.
Role type
Director, AI Automation Engineering (hands-on IC with team growth potential)
Builds
Automated testing infrastructure, evaluation frameworks, and CI/CD release gates for AI agents and LLM-based automation
Domain
Healthcare technology / Agentic AI / LLM automation
Deliverable
production ML models | infrastructure
Required skills
Python, pytest, CI/CD pipeline design, LLM output evaluation (RAG, prompt regression, safety, accuracy), agent lifecycle management, automated research chains, quality metrics definition
Preferred skills
Agent testing/orchestration platforms, LLM observability tooling (LangSmith, Braintrust), healthcare/pharma regulated environment experience, building QA functions from scratch
Technologies
GitHub Actions, Python, pytest, RAG, LLM eval frameworks
Responsibilities
Build infrastructure to continuously verify autonomous agent actions before and after production; Create automated evaluation frameworks for LLM outputs including accuracy, safety, and bias checks; Develop release-gating infrastructure for staged rollouts and rollback authority; Partner with AI engineers and architects to establish baseline quality metrics and validation standards; Document testing standards and runbooks for non-engineers to self-serve basic testing
Seniority
Director, hands-on IC with leadership growth
