AI Observability Engineer
Core
Ensuring AI solutions align with enterprise Responsible AI principles by monitoring model performance, detecting data bias, and managing AI governance programs.
Role type
AI Observability Engineer (Responsible AI & Governance)
Builds
Structured, trustworthy insights from AI system activities and executive-level reports on Responsible AI.
Domain
Insurance / Generative AI / Responsible AI
Deliverable
dashboards & analysis
Required skills
Machine Learning and LLM evaluation metrics, Data pipeline analysis, Responsible AI frameworks, Cross-functional collaboration, Regulatory framework translation, Complex problem solving
Preferred skills
Production LLM/RAG monitoring pipelines, OpenTelemetry distributed tracing, AI Observability tools (Fiddler, Arize, LangSmith), Dashboard/alert building
Technologies
LLMs, RAG, OpenTelemetry, Fiddler, Arize, LangSmith
Responsibilities
Track ML/LLM KPIs (accuracy, precision, recall, F1), Monitor data inputs for anomalies and bias, Manage AI model management platform operations, Review AI solutions for governance alignment, Create executive reports on Responsible AI, Develop RAI training content and assets, Monitor global AI regulatory frameworks
Seniority
Mid-level, hands-on IC
