Software Engineer, Web Crawling
On-siteVisa sponsorship offered
Core
Build massive-scale distributed web crawling infrastructure to process the entire web for an AI search engine.
Role type
Senior IC web crawler engineer
Builds
Distributed crawlers handling 100M+ pages/day, crawl scheduling systems, and anti-bot detection mechanisms
Domain
Internet infrastructure / Web crawling / AI search
Deliverable
production ML models
Required skills
High-performance language (C++, Rust), TypeScript, Playwright, Chrome DevTools Protocol (CDP), distributed systems design, rate limiting, JavaScript rendering, anti-bot detection, crawl scheduling algorithms
Preferred skills
Experience scaling web crawlers to Google-scale, optimizing system performance to exceptional degree
Technologies
C++, Rust, TypeScript, Playwright, Chrome DevTools Protocol