Senior Data Scientist AI Evaluation
Core
Designing and operationalizing evaluation frameworks to measure the correctness and quality of Alpaca's AI models and agents for financial services.
Role type
Senior individual contributor AI evaluation data scientist
Builds
Ground truth datasets, scoring methods, and automated evaluation loops for AI agents
Domain
Fintech / Brokerage infrastructure / AI evaluation
Deliverable
production ML models
Required skills
Statistical analysis, experimental design, Python, SQL, model validation, ground truth definition, metric design
Preferred skills
LLM-as-judge calibration, CI regression gates, text-to-SQL evaluation, fintech domain knowledge
Technologies
Python, SQL
Responsibilities
Define ground truth and metrics for AI models, build repeatable evaluation loops, partner with engineering to operationalize eval harnesses, translate eval results into system improvements, establish quality standards and documentation, mentor teams on evaluation best practices
Seniority
Senior, hands-on IC