Staff Software Engineer (Agentic Search) - Crawler
Core
Design and operate web-scale crawling and ingestion infrastructure to acquire, deduplicate, and refresh content from the open web and external data sources for an agent-native search platform.
Role type
Staff Software Engineer (Distributed Systems & Infrastructure)
Builds
Distributed crawling, scheduling, and ingestion systems processing billions of URLs
Domain
Cloud Infrastructure, AI Data Infrastructure, Web Scale Systems
Deliverable
infrastructure
Required skills
Go, C++, Distributed Systems, Web Protocols (HTTP, DNS, TLS), Content Extraction, Scalability, Fault Tolerance, Resource Management
Preferred skills
Kafka, Pulsar, NATS, RabbitMQ, Spark, Flink, Beam, MapReduce, Streaming Data Pipelines, Event-Driven Systems
Responsibilities
Design and implement web-scale crawling systems, Build ingestion workflows for internal and external data sources, Develop crawl scheduling and prioritization policies, Build systems for URL discovery and deduplication, Ensure reliable operation under high-throughput conditions, Define observability and quality metrics for crawl coverage
