Staff / Senior Software Engineer (Agentic Search) - Crawler
Core
Building distributed crawling and ingestion infrastructure for an agent-native search platform that provides verified real-world data to AI systems.
Role type
Senior Software Engineer (Distributed Systems / Crawler Infrastructure)
Builds
Web-scale crawling systems, ingestion workflows, and URL discovery/orchestration pipelines for AI agents.
Domain
AI Infrastructure / Search Engines / Distributed Systems
Deliverable
infrastructure
Required skills
Go or C++, large-scale distributed systems, web protocols (HTTP, DNS, TLS), content extraction, observability, resource management
Preferred skills
Kafka/Pulsar/NATS/RabbitMQ, streaming data pipelines, Spark/Flink/Beam, ad tech or search engine experience
Responsibilities
Design and operate web-scale crawling systems, build ingestion workflows for internal/external sources, develop crawl scheduling and prioritization policies, ensure reliable operation under high-throughput, define observability metrics, monitor resource usage and costs, collaborate with indexing/ML teams.
Seniority
Senior, hands-on IC
