Senior Member of Technical Staff, Web Data
Core
Building large-scale web data pipelines to transform raw internet data into high-quality training corpora for pre-training frontier language models.
Role type
Senior IC data pipeline engineer (web data)
Builds
Large-scale data pipelines for pre-training AI models
Domain
AI/ML, Natural Language Processing, Web Data Engineering
Deliverable
production ML models
Required skills
Python, data pipeline development, Apache Spark, Apache Beam, Pandas, large-scale dataset processing, data quality assessment, experimentation with data mixtures
Preferred skills
Experience with web corpora, bridging research and engineering
Technologies
Python, Apache Spark, Apache Beam, Pandas
Responsibilities
Maintain large-scale pipelines for processing web corpora, develop filtering and quality-scoring systems, analyze web data composition across domains and time periods, develop and maintain deduplication pipelines, collaborate with cross-functional teams on data pipeline demands
Seniority
Senior, hands-on IC