Assistant Director Data Scientist
Core
Evaluate and validate large language models (LLMs) for production-grade analytical and decision-support systems in credit analytics.
Role type
Assistant Director Data Scientist (LLM Evaluation)
Builds
Evaluation frameworks, metrics, and benchmarks for LLM performance in credit risk applications.
Domain
Financial Services / Credit Analytics / Generative AI
Deliverable
production ML models
Required skills
Statistical methods, experimental design, hypothesis testing, Python or R programming, LLM evaluation benchmarks, model validation methodologies
Preferred skills
LLM/generative AI system evaluation, production ML systems experience, cloud platforms (AWS, GCP, Azure), publications in model evaluation
Technologies
Python, R, AWS, Azure, GCP, LLM
Responsibilities
Design and build evaluation frameworks to assess LLM performance in credit analytics, develop metrics to measure model robustness and reliability, analyze model behavior to identify failure modes and edge cases, partner with development teams to embed validation processes, assess model outputs for bias and fairness, create documentation for evaluation methods and findings
Seniority
Senior, hands-on IC with leadership responsibilities