Agentic AI Optimization Developer
Core
Build automated evaluation frameworks and guardrails to ensure the safety, reliability, and accuracy of autonomous multi-agent AI systems in production.
Role type
Agentic AI Evaluation & Tuning Engineer (LLMOps)
Builds
Automated evaluation pipelines, golden datasets, and production guardrail frameworks for multi-agent architectures.
Domain
Enterprise AI / Autonomous Workflows / LLM Orchestration
Deliverable
production ML models
Required skills
LLM orchestration frameworks, function calling & JSON schema design, automated test scripting (Python), asynchronous system debugging, RAG pipeline optimization, prompt tuning, multi-agent trajectory auditing
Preferred skills
AI evaluation platforms, GenAI production testing, Google Cloud AI suite, UiPath ecosystem, synthetic data generation, TypeScript, microservices architecture
Technologies
LangGraph, Python, TypeScript, Google Cloud Vertex, Gemini Enterprise, UiPath Maestro
Responsibilities
Curate and maintain high-quality golden datasets for agent testing; Design and implement continuous automated evaluation pipelines; Trace and audit multi-step agent reasoning processes; Optimize system prompts and tool/function-calling interactions; Monitor deployed agents to mitigate drift, hallucinations, and prompt injections; Optimize RAG knowledge bases for accurate data retrieval
Seniority
Mid-Senior, hands-on IC
