Senior Specialist, AI Evals Engineer
Core
Design, build, and execute evaluations of AI systems (foundation models, RAG, agents) to assess quality, accuracy, robustness, and safety for a global healthcare biopharma company.
Role type
Senior AI Evals Engineer (hands-on IC with technical leadership)
Builds
Scalable, reusable evaluation pipelines, tooling, and datasets integrating into AI engineering delivery
Domain
Healthcare biopharma + Generative AI / LLMs / RAG / Agentic AI
Deliverable
production ML models
Required skills
AI/ML system evaluation & testing, LLM evaluation (LLM-as-a-judge), RAG evaluation, agentic AI evaluation, AI safety testing/red teaming, Python (scientific computing stack), evaluation dataset development, metrics & scoring rubrics design, statistical analysis
Preferred skills
AI evaluation platforms (DeepEval, Ragas, LangSmith, etc.), AI security/red-teaming tools (SPLX, Mindgard, etc.), AI governance tools (Credo AI, IBM AIF360), cloud infrastructure (AWS/Azure), pharmaceutical domain knowledge
Technologies
Python, LLMs, RAG pipelines, Agentic AI frameworks, AWS, Azure
Responsibilities
Lead design and execution of AI system evaluations across the stack; define evaluation datasets, test suites, and metrics; own evaluation strategy and architecture; analyze results and communicate findings to stakeholders; partner with engineering and risk teams to set standards; provide technical leadership and mentorship to evaluation engineers; stay current with AI evaluation research and drive adoption
Seniority
Senior, hands-on IC with technical leadership
