Software Engineer - Scraping Platform
Core
Design and evolve a distributed scraping and navigation platform to resiliently recover information at scale using AI models and agents.
Role type
Senior IC backend software engineer (distributed systems & scraping)
Builds
A resilient, scalable distributed scraping platform serving Mercado Libre's data recovery needs
Domain
E-commerce / Financial Services / Data Engineering
Deliverable
production ML models
Required skills
Python, Go, Node.js, distributed systems, asynchronous architectures, event-oriented systems, message queues, concurrent processing, SQL, streams, object storage, browser automation, proxy management, IP rotation, anti-bot mechanisms, observability, incident resolution, root cause analysis, technical architecture, code reviews, mentorship
Preferred skills
GenAI, agile learning, comfort with ambiguity
Technologies
Python, Go, Node.js, SQL, message queues, object storage, browser automation tools, proxy networks
Responsibilities
Design and evolve a distributed scraping platform handling variable loads with idempotency, retries, and circuit breakers; Automate browser navigation and manage sessions, proxies, and anti-bot mechanisms; Integrate AI models and agents for dynamic navigation and data recovery decisions; Own critical production components from design to incident resolution and capacity planning; Lead architectural decisions and mentor team members
Seniority
Senior, hands-on IC with mentorship responsibilities