Research Engineer, Web Crawling
Core
Build and own web-crawling systems for distributed collection, filtering, and deduplication of internet-scale data to fuel AI pretraining.
Role type
Senior IC software engineer (distributed systems)
Builds
Production web crawlers, ingestion infrastructure, and data pipelines for pretraining data
Domain
Internet-scale data acquisition and AI pretraining
Deliverable
production ML models
Required skills
Python, Go, Rust, distributed systems, web crawling, data pipelines, petabyte-scale storage, technical direction
Preferred skills
Machine learning for crawl selection, open-source contributions, search engine background
Technologies
Python, Go, Rust
Responsibilities
Design and scale web crawler and ingestion infrastructure; Build pipelines for extraction, deduplication, and data quality filtering; Build specialized crawlers for hard-to-reach data sources; Work with pretraining team to correlate data changes with model performance; Improve reliability and efficiency of crawling infrastructure at petabyte scale; Set technical direction and mentor engineers
Seniority
Senior, hands-on IC