Software Engineer, Web Crawling
Core
Build massive-scale distributed web crawling infrastructure to process the entire web for an AI search engine.
Role type
Senior IC web crawler engineer
Builds
Distributed crawlers handling 100M+ pages per day
Domain
Internet-scale data ingestion and crawling
Deliverable
production ML models
Required skills
C++, Rust, TypeScript, Playwright, Chrome DevTools Protocol (CDP), distributed systems design, rate limiting, dynamic content rendering, anti-bot detection, crawl scheduling algorithms
Preferred skills
Experience scaling web crawlers to Google-scale, optimizing system performance to exceptional degree
Technologies
C++, Rust, TypeScript, Playwright, CDP
Responsibilities
Build distributed crawlers handling 100M+ pages per day, Optimize crawl politeness and rate limiting across thousands of domains, Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures, Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency
Seniority
Senior, hands-on IC