Senior Software Engineer (Web Scraping, Data acquisition, Data collection)
Core
Design and build distributed web scraping architectures to collect data from thousands of domains and mobile apps for top retailers and brands.
Role type
Senior Software Engineer (Web Scraping & Data Acquisition)
Builds
Scalable distributed crawlers, data pipelines, and monitoring systems for consumer intelligence.
Domain
Retail analytics, data acquisition, anti-bot evasion.
Deliverable
production ML models | product features
Required skills
Python, distributed system architecture, API reverse engineering, anti-bot evasion techniques, proxy management, headless browser automation, debugging complex HTTP/browser interactions, structured data storage (Parquet/Avro), database optimization (MongoDB/InfluxDB/Redis).
Preferred skills
Real-device mobile simulation, distributed task queues (Celery/Kafka), high-security platform scraping, Linux/Cloud environments (AWS/GCP), container orchestration (Kubernetes/Cloud Run).
Responsibilities
Design scalable distributed scraping architectures; reverse engineer APIs and mobile apps; detect and bypass anti-bot measures; lead technical innovation and prototype new approaches; set up anomaly detection and monitoring for scraping failures.
Seniority
Senior, hands-on IC
