Data Infrastructure Engineer, Pre-training
Core
Design and implement highly performant, reproducible, and traceable data-processing infrastructure for large language model training.
Role type
Senior IC data infrastructure engineer (LLM pre-training)
Builds
End-to-end pipelines transforming raw web-scale corpora into training-ready datasets
Domain
Artificial Intelligence / Large Language Models / Distributed Systems
Deliverable
production ML models
Required skills
Distributed computing frameworks, Apache Spark, Python, Rust, fault-tolerant system design, data quality assurance, scalable processing primitives
Preferred skills
Language model training infrastructure, MLOps, ML infrastructure, advanced degree in Computer Science
Technologies
Apache Spark, Python, Rust
Responsibilities
Develop scalable processing primitives including tokenization, deduplication, and chunking; Build data quality assurance and validation systems at scale; Collaborate with research teams to implement novel data-processing architectures; Develop distributed computing architectures and fault-tolerant infrastructure components
Seniority
Senior, hands-on IC
