Staff Software Engineer (Agentic Search) - Crawler
Core
Design and operate web-scale crawling and ingestion infrastructure to acquire, deduplicate, and refresh content from the open web and external sources for an agent-native search platform.
Role type
Staff Software Engineer (Distributed Systems & Crawler Infrastructure)
Builds
Distributed crawling, scheduling, and ingestion systems processing billions of URLs
Domain
Internet-scale data infrastructure, AI search, distributed systems
Deliverable
production ML models
Required skills
Go, C++, distributed systems design, web protocols (HTTP, DNS, TLS), high-throughput pipelines, fault tolerance, resource management
Preferred skills
Kafka, Pulsar, NATS, RabbitMQ, Spark, Flink, Beam, MapReduce, streaming data pipelines, event-driven systems
Responsibilities
Design and implement web-scale crawling systems, build ingestion workflows for internal/external data sources, develop crawl scheduling and prioritization policies, build URL discovery and deduplication systems, ensure reliable operation under high-throughput, define observability metrics, monitor resource usage and costs, collaborate with indexing and ML teams.
Seniority
Staff, hands-on IC
