AI Reliability Manager
Core
Ensure the accuracy, trustworthiness, and continuous improvement of agentic AI capabilities in an investigative platform for law enforcement and financial crimes compliance.
Role type
Senior individual contributor AI reliability manager
Builds
Evaluation data, quality signals, and release readiness assessments for agentic AI systems
Domain
AI safety and reliability in investigative/fraud detection software
Deliverable
production ML models | dashboards & analysis
Required skills
Analytical rigor, pattern recognition in AI output, evaluation of open-ended work, hands-on generative AI tool usage, technical writing
Preferred skills
Domain knowledge in law enforcement/criminal investigations or AML/BSA, understanding of retrieval-based AI failure modes (hallucination, citation mismatch)
Responsibilities
Own and extend gold datasets and scoring rubrics for AI evaluation, execute evaluation cycles and analyze failure patterns, validate AI features prior to release, manage engineering tickets for AI output issues, partner with product and engineering to translate findings into roadmap decisions
Seniority
Senior, hands-on IC
