CareerPlan

Research Engineer, Web Crawling

San FranciscoFull-time$350k–$475k2026-09-24 → 2026-11-19
On-siteVisa sponsorship offered

Core

Build and own web-crawling systems for distributed collection, filtering, and deduplication of internet-scale data to fuel AI pretraining.

Role type

Senior IC software engineer (distributed systems)

Builds

Production web crawlers, ingestion infrastructure, and data pipelines for pretraining data

Domain

Internet-scale data acquisition and AI pretraining

Deliverable

production ML models

Required skills

Python, Go, Rust, distributed systems, web crawling, data pipelines, petabyte-scale storage, technical direction

Preferred skills

Machine learning for crawl selection, open-source contributions, search engine background

Technologies

Python, Go, Rust

Responsibilities

Design and scale web crawler and ingestion infrastructure; Build pipelines for extraction, deduplication, and data quality filtering; Build specialized crawlers for hard-to-reach data sources; Work with pretraining team to correlate data changes with model performance; Improve reliability and efficiency of crawling infrastructure at petabyte scale; Set technical direction and mentor engineers

Seniority

Senior, hands-on IC