Staff Data Scientist, AI Evaluations Platform
Core
Senior technical leader responsible for designing and building evaluation datasets, LLM judge methods, deterministic scorers, and measurement frameworks to measure model and agent quality, safety, and performance for RBC's enterprise AI systems.
Role type
Staff Data Scientist (AI Evaluations)
Builds
Evaluation datasets, scorecards, LLM-as-judge methods, deterministic scorers, human evaluation protocols, and measurement frameworks for generative AI and agentic systems.
Domain
Financial services, Generative AI, Agentic AI, Model Risk Management
Deliverable
production ML models
Required skills
Data science, Applied machine learning, AI evaluation methodologies, LLM evaluation (LLM-as-judge, deterministic scoring, human evaluation), Statistics, Data curation, Python, SQL, Technical leadership and mentorship
Preferred skills
Agentic AI evaluation, Financial services/regulated AI experience, MLflow/Langfuse/LangSmith/OpenTelemetry/Grafana, Publications or patents
Technologies
Python, SQL, MLflow, Langfuse, LangSmith, OpenTelemetry, Grafana
Responsibilities
Design advanced model and agent evaluation methodologies; Define evaluation science standards; Own end-to-end lifecycle for evaluation datasets and scorecards; Design scalable evaluation approaches for generative AI and agentic systems; Partner with cross-functional teams to embed evaluations into workflows; Establish human evaluation and review protocols; Measure and improve scorer accuracy and robustness; Provide technical leadership and mentorship to junior data scientists.
Seniority
Staff, Senior technical leader with mentorship responsibilities