Business Analyst II
Core
Design and manage quality measurement frameworks, evaluation programs, and ground-truth creation for AI-powered Trust & Safety systems to improve model performance and safety outcomes.
Role type
Senior IC Business Analyst (AI Quality & Evaluation)
Builds
Quality measurement frameworks, evaluation programs, ground-truth datasets, and scalable operational processes for AI systems.
Domain
Trust & Safety, AI/ML Operations, Marketplace Integrity
Deliverable
production ML models | product features
Required skills
Designing evaluation methodologies, creating ground-truth datasets, defining labeling strategies, performing hands-on labeling and adjudication, generating actionable insights from operational data, translating ambiguous quality challenges into scalable frameworks, partnering with Engineering and Data Science teams, leveraging data to identify quality issues, applying Generative AI and LLM concepts, prototyping solutions, building proofs of concept.
Preferred skills
Experience with Python, SQL, Power BI, Azure AI services, prompt engineering, RAG, workflow automation, agentic frameworks, transforming manual workflows into automation-first solutions.
Technologies
Python, SQL, Power BI, Power Platform, Azure AI services
Responsibilities
Partner with Engineering, Data Science, and Policy teams to define quality frameworks; Design and manage evaluation programs and calibration processes; Perform hands-on labeling, content review, and adjudication to establish ground truth; Establish quality metrics and scorecards to measure reviewer and model effectiveness; Generate insights from datasets to identify risks and improvement opportunities; Drive cross-functional initiatives from problem identification to execution.
Seniority
Senior, hands-on IC
