Data Infra - Signals and Evaluations
Core
Build versioned workflows, benchmark suites, and simulation infrastructure for ML evaluation, dataset engineering, and agent training across perception, reasoning, and planning domains.
Role type
Senior IC data infrastructure engineer (ML evaluation & simulation)
Builds
Versioned dataset workflows, benchmark suites, simulation environments, and automated promotion gates for AI systems.
Domain
Artificial Intelligence / Machine Learning / Data Infrastructure
Deliverable
production ML models
Required skills
ML evaluation, dataset engineering, reinforcement-learning environments, simulation, scientific ML, test infrastructure, statistics, experimental design, benchmark validity, distribution shift, calibration, reward design, Python, software engineering discipline
Preferred skills
dataset curation, benchmark platforms, deep reinforcement learning, world models, synthetic data, multimodal evaluation, scientific validation, PyTorch, JAX, distributed evaluation, experiment tracking, environment frameworks, model and agent observability, uncertainty quantification, calibration, out-of-distribution detection, conformal methods, selective prediction, robotics, industrial systems
Responsibilities
Build versioned workflows for dataset collection, curation, filtering, labeling, ground truth, synthetic data, hard-negative mining, and contamination controls. Design benchmark suites for perception, multimodal reasoning, physics-informed prediction, retrieval, planning, tool use, and world modeling. Create simulation and environment infrastructure for reinforcement learning, imitation learning, offline learning, and agent training. Define repeatable evaluations covering accuracy, calibration, generalization, robustness, safety, latency, and cost. Convert field, mission, customer, and experimental failures into versioned datasets, benchmark cases, adversarial tests, and environment scenarios. Build automated promotion gates and communicate tested capabilities, unresolved gaps, and release evidence.
Seniority
Senior, hands-on IC
