Product & Research Operations Manager, Human Data
Core
Design, scale, and operate a global workforce of human evaluators to support AI model training and evaluation pipelines.
Role type
Product & Research Operations Manager (Human Data)
Builds
Production systems for human-in-the-loop evaluation, including workforce structures, vendor relationships, and quality control workflows.
Domain
AI/ML data operations, specifically for speech and language model evaluation.
Deliverable
production ML models
Required skills
Workforce management, vendor negotiation, capacity modeling, quality assurance design, inter-rater reliability analysis, operational scaling, systems thinking, data analysis.
Preferred skills
AI/ML data operations experience, audio/speech workflow knowledge, QA system familiarity, marketplace platform experience, multilingual operations.
Technologies
Upwork, Mercor, annotation tooling.
Responsibilities
Design workforce structure across languages and skill tiers; build capacity models for evaluation pipelines; manage vendor relationships and negotiate SLAs; decide on build/buy/hybrid workforce models; design multi-layer QA systems; track inter-rater reliability and error rates; build escalation and retraining workflows; run day-to-day task allocation and throughput tracking; partner with tooling and data teams to improve UX and output quality.
Seniority
Mid-Senior, hands-on IC
