AI Engineer (Evaluation)
Core
Build evaluation frameworks and pipelines to measure the capabilities of Large Language Models (LLMs), specifically focusing on multilingual, multicultural, and multimodal aspects.
Role type
AI Engineer (Evaluation)
Builds
Evaluation frameworks, pipelines, and datasets for LLMs
Domain
Artificial Intelligence / Large Language Models
Deliverable
production ML models
Required skills
LLM evaluation experimental design, LLM inference frameworks (vLLM), Deep Learning frameworks (PyTorch), Python, Git, data preparation and analysis, AI modelling, coding, testing, validation, deployment
Preferred skills
Reading and understanding research papers, community engagement
Technologies
vLLM, PyTorch, Python, Git
Responsibilities
Develop and maintain evaluation frameworks and pipelines; experiment with latest research in multilingual, multicultural, and multimodal LLM evaluations; collect, translate and verify evaluation datasets; perform data preparation, analysis, modelling, coding, testing, validation and deployment; collaborate with cross-functional teams; maintain code repository and documentation standards
Seniority
Mid-level, hands-on IC