Ai Product Analyst Ai Receptionist Team

## About the role At MEDvidi, we are scaling AI-powered systems across critical parts of our product and operations. As these systems become more autonomous and deeply integrated into user workflows, understanding how they behave in production becomes a core business challenge. We are hiring a Product Analyst to help us evaluate, monitor, and improve AI agents and LLM-powered workflows operating at scale. This role is focused on analyzing real AI system behavior in production environments: - understanding product and operational impact, - identifying failure patterns on generational tasks, - measuring quality and reliability of outputs, - helping teams make accurate decisions under ambiguity. ## Responsibilities - Analyze production behavior of AI agents, automation systems, and LLM-powered workflows, - Investigate hallucinations, inconsistencies, failure modes, and quality degradation across automated user journeys, - Work directly with logs, conversations, raw system outputs, and operational datasets to identify hidden patterns and actionable insights, - Design evaluation approaches and analytical frameworks for measuring AI system quality and reliability, - Build monitoring systems and dashboards for AI performance visibility, - Identify operational and product risks introduced by automation, - Measure the impact of AI systems on business metrics, operational efficiency, and user experience, - Detect data quality issues, shifts, and anomalies affecting AI system performance, - Partner closely with Product, Engineering, QA, and Operations teams on AI-related initiatives, - Support rapid decision-making in fast-changing and ambiguous environments. ## Requirements - 3+ years of experience in analytical roles related to AI systems, ML products, automation platforms, or complex production systems, - Strong Python skills for deep analytical investigations and working with large-scale noisy text data, - Advanced SQL skills, - Understanding of LLM limitations, hallucinations, evaluation challenges, prompt sensitivity, model trade-offs, and AI failure modes, - Experience building monitoring systems, analytical frameworks, or operational dashboards for production systems, - Ability to work with incomplete, ambiguous, and rapidly changing data environments, - Strong ownership mindset and ability to independently drive investigations and analytical conclusions, - Ability to connect AI system quality with business and operational impact, - Fluent Russian; English level B2 or higher, - Ability to work CET-overlapping hours. ## Nice to have - Experience with conversational AI or AI agents, - Experience evaluating LLM outputs or annotation quality, - Experience with AI quality metrics and evaluation methodologies, - Experience working with Langchain infrastructure, - Experience with ML data pipelines or synthetic data generation, - Background in anomaly detection. ## What we offer - Work on a modern AI product where quality is a strategic function, not a formality - Real influence on product direction and engineering practices - High-impact work in a product-driven medical tech company - Startup environment: fast iteration, high ownership, minimal bureaucracy - Competitive compensation and growth opportunities ## Benefits - Health insurance after probation period - Sports & wellness compensation - Fully remote, long-term B2B collaboration - 19 working days of vacation per year - 3 additional wellness days per year - Paid sick leave for the first 5 working days - Gifts to employees for important events - Career Development - Personalized English lessons via Preply Ready to make an impact? Send us your profile - let's build something meaningful together.