Specialist, AI Evals Engineer
Core
Design, build, and run evaluations of AI systems across the stack (foundation models, RAG pipelines, agentic applications) to assess quality, accuracy, robustness, and safety for a global healthcare biopharma company.
Role type
Specialist AI Evals Engineer
Builds
Evaluation datasets, test suites, metrics, scoring rubrics, and reusable evaluation pipelines integrating into AI engineering delivery.
Domain
Healthcare biopharma / Generative AI
Deliverable
production ML models
Required skills
GenAI, NLP, Machine Learning, AI Model Evaluation & Testing, RAG Evaluation, Agentic AI Evaluation, AI Evaluation Frameworks & Benchmarks, LLM-as-a-Judge Methodologies, AI Safety Testing / Red Teaming, Python Programming, Evaluation Dataset Creation, Metrics & Scoring Rubrics Development, Evaluation Pipeline Development, Statistical Analysis
Preferred skills
AI Testing & Evaluation Tools, AI Security & Red Teaming Tools, AI Governance & Responsible AI Tools, Qualitative & Quantitative Research Methods, Cloud Platforms (AWS / Microsoft Azure), Pharmaceutical / Life Sciences Domain Knowledge, Bioinformatics, Statistics, Mathematics, Physics background
