Senior Software Engineer (Web Scraping, Data acquisition, Data collection)
Core
Design and build distributed web scraping architectures to collect data from thousands of domains and mobile apps, bypassing anti-bot measures to provide actionable insights for retailers.
Role type
Senior IC software engineer (web scraping & data acquisition)
Builds
Scalable distributed crawlers, data pipelines, and monitoring systems for high-volume data collection
Domain
E-commerce, Retail, Consumer Intelligence
Deliverable
production ML models
Required skills
Python, distributed system architecture, API reverse engineering, anti-bot evasion, proxy management, headless browser automation, debugging, database optimization
Preferred skills
Real-device simulation, distributed task queues (Celery, Kafka), cloud orchestration (Kubernetes, AWS, GCP), structured data formats (Parquet, Avro)
Technologies
Frida, mitmproxy, Charles Proxy, Burp, Scrapy, Requests, httpx, Selenium, Playwright, Celery, Kafka, MongoDB, InfluxDB, Redis, AWS, GCP, Kubernetes
Responsibilities
Design distributed scraping architectures, reverse engineer APIs and mobile apps, detect and bypass anti-bot measures, lead technical innovation and prototyping, set up anomaly detection and monitoring
Seniority
Senior, hands-on IC
