Junior Web Crawling

## About the Role We are looking for a proactive and detail-oriented individual with strong problem-solving skills, a curious mindset, and the ability to take ownership in a fast-paced environment, along with good communication skills. ## Key Responsibilities - Design, develop, and maintain scalable web crawlers using Scrapy, Selenium, Playwright, and custom frameworks - Handle dynamic websites including JavaScript-heavy pages, pagination, and anti-bot mechanisms - Build and optimize high-throughput data pipelines using RabbitMQ / Kafka - Ensure data quality, consistency, and reliability across workflows - Work with large datasets in MongoDB and distributed systems - Troubleshoot crawler issues, performance bottlenecks, and data inconsistencies - Collaborate with data and analytics teams for structured data delivery - Improve crawling efficiency, success rates, and system scalability ## Requirements - Strong proficiency in Python - Hands-on experience with Scrapy, Selenium, or Playwright - Experience with messaging systems (RabbitMQ, Kafka) - Working knowledge of MongoDB and PostgreSQL - Understanding of HTTP/HTTPS, HTML, DOM, XPath, and CSS selectors - Familiarity with proxies, rate limiting, session handling, and CAPTCHA handling ## What We're Looking For - Strong problem-solving skills - Curiosity to analyze and reverse-engineer websites - Ownership and accountability - Ability to work in a fast-paced environment - Good communication skills ## About the Company 1DigitalStack.ai is a cutting-edge Technology and Data Science product company helping brands win and grow profitably on e-commerce marketplaces across the globe. Our platforms empower customers to accelerate revenue growth through deep e-marketplace data, advanced and custom analytics, actionable intelligence, and end-to-end media optimization and automation. With our solutions, Brand Managers, P&L Owners, E-commerce Managers, Channel & Category Managers, and Marketing Leaders unlock new revenue opportunities every day. We partner with some of India's largest consumer brands, including Unilever, Marico, Coke, Unicharm, Tata Consumer, and Dabur. We operate across 220+ global marketplaces spanning Southeast Asia, Europe, and the UAE, enabling enterprises to maximize performance, profitability, and scale on e-commerce channels.