CareerPlan

Senior Machine Learning Engineer, Evals

AmsterdamFull-time2026-11-16 → 2026-11-20
Hybrid

Core

Design, curate, and operate evaluation frameworks and benchmarks to measure the performance of the Large Commerce Model (LCM) and guide its training, deployment, and research decisions.

Role type

Senior Machine Learning Engineer (Evaluation & Benchmarking)

Builds

Evaluation platform, evaluation tasks, datasets, graders, and public commerce benchmarks for the LCM foundation model.

Domain

E-commerce, Fintech, Large Language Models (LLMs), Foundation Models

Deliverable

production ML models

Required skills

Python, PyTorch, Transformers, experimental design, statistics, large-scale inference debugging, LLM-as-judge calibration, agent evaluation, data curation, version control, infrastructure automation

Preferred skills

Language-model training or post-training, agent or gaming environments, human evaluation, recommendation metrics, online experimentation, public benchmark contributions

Technologies

Python, PyTorch, Transformers, LLMs, GPU infrastructure

Responsibilities

Design evaluations with scientists and domain experts; curate evaluation data and calibrate task difficulty; own the development and operation of the evaluation platform; develop programmatic checks and LLM-based graders; analyze model outputs and agent trajectories; maintain versioned tasks and environments; prepare commerce benchmarks for public release; integrate evaluations into training and release workflows; build tools to generate tasks and investigate failures.

Seniority

Senior, hands-on IC