Senior AI Product Ops
Core
Own end-to-end production monitoring for AI agents, ensuring reasoning correctness, context currency, and behavior governance as they learn in real-time.
Role type
Senior AI Product Operations Engineer (AI Ops)
Builds
Production observability dashboards, evaluation frameworks, and governance workflows for agentic systems.
Domain
Financial technology / AI Operations / LLMs
Deliverable
production ML models
Required skills
AI/ML production monitoring, observability, evaluation engineering, drift detection, root cause analysis, governance, compliance
Preferred skills
Responsible AI practices, PII detection, audit logging, policy enforcement
Technologies
Tracing, telemetry, guardrails, automated evals, regression testing
Responsibilities
Track key metrics (KPIs, accuracy, latency, SLA breaches, guardrail triggers) via observability dashboards; diagnose issues by inspecting traces and classifying root causes (intent, tool, parameter, hallucination); evaluate agent reasoning correctness and learning loop drift; manage post-production reliability for API and external endpoint failures; partner with AI Engineering to validate prompt/context changes via evals; embed approval workflows and audit logging into the agent's learning loop.
Seniority
Senior, hands-on IC
