Data Infrastructure Engineer, Pre-training
Core
Design and implement high-throughput, fault-tolerant distributed data processing infrastructure for training large language models.
Role type
Staff-level Data Infrastructure Engineer (Pre-training)
Builds
End-to-end data pipelines transforming web-scale corpora into training-ready datasets for LLMs
Domain
Artificial Intelligence / Large Language Model Training Infrastructure
Deliverable
production ML models
Required skills
Distributed systems engineering, Apache Spark, Python, Rust, Data quality assurance, Scalability design, Fault tolerance
Preferred skills
Experience with language model training infrastructure, System reliability optimization, Ambiguous requirement navigation, Independent problem ownership
Technologies
Apache Spark, Python, Rust
Responsibilities
Design and implement data processing infrastructure for LLM training; Develop core processing primitives (tokenization, deduplication, chunking); Build robust data quality assurance and validation systems; Collaborate with research teams on novel data architectures; Operate end-to-end data pipelines for web-scale corpora
Seniority
Staff, hands-on IC with strategic impact
