Senior Machine Learning Engineer, Evals
Core
Design, curate, and operate evaluation frameworks and benchmarks to measure the performance of the Large Commerce Model (LCM) and guide its training, deployment, and research decisions.
Role type
Senior Machine Learning Engineer (Evaluation & Benchmarking)
Builds
Evaluation platform, evaluation tasks, datasets, graders, and public commerce benchmarks for the LCM foundation model.
Domain
E-commerce, Fintech, Large Language Models (LLMs), Foundation Models
Deliverable
production ML models
Required skills
Python, PyTorch, Transformers, experimental design, statistics, large-scale inference debugging, LLM-as-judge calibration, agent evaluation, data curation, version control, infrastructure automation
Preferred skills
Language-model training or post-training, agent or gaming environments, human evaluation, recommendation metrics, online experimentation, public benchmark contributions
Technologies
Python, PyTorch, Transformers, LLMs, GPU infrastructure
Responsibilities
Design evaluations with scientists and domain experts; curate evaluation data and calibrate task difficulty; own the development and operation of the evaluation platform; develop programmatic checks and LLM-based graders; analyze model outputs and agent trajectories; maintain versioned tasks and environments; prepare commerce benchmarks for public release; integrate evaluations into training and release workflows; build tools to generate tasks and investigate failures.
Seniority
Senior, hands-on IC