Senior Data Engineer, Training Data Platform
Core
Build and operate a training-data platform for a large commerce foundation model (LCM), covering data acquisition, processing, synthetic generation, quality evaluation, and dataset versioning.
Role type
Senior IC data engineer (LLM training data platform)
Builds
Reproducible training datasets, distributed processing workflows, and automated data quality/agent tools for a foundation model.
Domain
AI/ML infrastructure, large-scale data engineering, commerce data
Deliverable
production ML models
Required skills
Python, SQL, distributed processing (Spark/Ray/Beam), cloud storage, workflow orchestration, data quality, versioning, lineage
Preferred skills
LLM datasets, commerce data, content extraction, synthetic data generation, agents, AWS/Databricks/EMR
Technologies
Spark, Ray, Beam, AWS, Databricks, EMR
Responsibilities
Build acquisition and processing pipelines for portfolio and external data; Develop shared platform for inspecting sources and publishing datasets; Implement dataset versioning and lineage; Build quality checks for completeness, freshness, and accuracy; Build synthetic data pipelines for pre- and post-training; Build agents to turn catalogue sources into training datasets.
Seniority
Senior, hands-on IC