Team8- Fintech Stealth Startup- Senior AI Researcher, Evaluation & Agent Performance
Core
Design and build evaluation harnesses, benchmarks, and metrics to assess the performance, reliability, and efficiency of AI agents operating on code and runtime data.
Role type
Senior AI Researcher (Evaluation & Agent Performance)
Builds
Eval harnesses, benchmarks, regression suites, and performance metrics for AI agents
Domain
Fintech / Software Engineering / AI Agents
Deliverable
production ML models
Required skills
LLM evaluation, agent architecture design, benchmark creation, statistical analysis, Python, CI/CD integration, research leadership
Preferred skills
LLM-as-judge methods, repo-level code benchmarks, distributed systems, observability data
Technologies
Python, CI, MCP, LLMs
Responsibilities
Design and build eval harnesses and benchmarks; Define and track agent performance metrics; Run controlled comparisons across models and strategies; Diagnose agent failures and drive engineering fixes; Build regression suites in CI; Lead external research and publish papers; Set the research agenda and mentor hires
