Data Scientist
Core
Define and track metrics, design A/B tests, and analyze data to measure the impact of AI agents and product features on real-world client outcomes.
Role type
Senior Product Data Scientist (AI/ML evaluation)
Builds
Dashboards, self-serve tools, and analytical frameworks for product and ML teams
Domain
AI infrastructure, autonomous agents, enterprise operations
Deliverable
dashboards & analysis
Required skills
experimental design, A/B testing, causal inference, SQL, Python, product metrics definition, root-cause analysis, statistical modeling
Preferred skills
LLMs, AI agents, generative AI, probabilistic ML, offline-to-online evaluation, unstructured data analysis, enterprise B2B products
Technologies
SQL, Python, data warehouses, experimentation platforms, business intelligence tools
Responsibilities
Define and track product, feature, and agent-level metrics; Design, run, and interpret A/B tests for model changes and agent behavior; Measure agent performance impact on client outcomes; Connect offline model evaluations with production performance; Conduct deep analyses across conversations and workflows; Investigate anomalies and perform root-cause analyses; Build statistical models and analytical frameworks; Partner with Engineering to improve instrumentation and data quality; Build dashboards and self-serve tools; Communicate findings to stakeholders
Seniority
Senior, hands-on IC
