CareerPlan

Software Engineer, Web Crawling

San Francisco, CaliforniaFull-time2025-07-23 → 2026-11-19
On-siteVisa sponsorship offered

Core

Build massive-scale distributed web crawling infrastructure to process the entire web for an AI search engine.

Role type

Senior IC web crawler engineer

Builds

Distributed crawlers handling 100M+ pages per day

Domain

Internet-scale data ingestion and crawling

Deliverable

production ML models

Required skills

C++, Rust, TypeScript, Playwright, Chrome DevTools Protocol (CDP), distributed systems design, rate limiting, dynamic content rendering, anti-bot detection, crawl scheduling algorithms

Preferred skills

Experience scaling web crawlers to Google-scale, optimizing system performance to exceptional degree

Technologies

C++, Rust, TypeScript, Playwright, CDP

Responsibilities

Build distributed crawlers handling 100M+ pages per day, Optimize crawl politeness and rate limiting across thousands of domains, Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures, Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency

Seniority

Senior, hands-on IC