Senior Web Scraper (Scrapy)
Core
Design and build distributed scraping architectures to extract high-quality, actionable data at scale from thousands of web and mobile domains.
Role type
Senior IC web scraper engineer (distributed systems)
Builds
Distributed scraping infrastructure and crawlers for consumer intelligence
Domain
Data extraction / Web scraping / E-commerce intelligence
Deliverable
production ML models
Required skills
Python, Scrapy, reverse engineering APIs/mobile apps, anti-bot evasion, proxy management, session/cookie handling, TLS tweaking, distributed task queues, cloud orchestration
Preferred skills
Real-device simulation, high-security platform scraping, Linux, Kubernetes, real-time stream processing
Responsibilities
Design distributed scraping architectures, reverse engineer APIs and mobile apps, detect and bypass anti-bot measures, lead technical innovation in scraping, set up anomaly detection and monitoring
Seniority
Senior, hands-on IC
