Senior AI Engineer (f/m/d) - Remote in Germany
Core
Design, build, and evaluate agentic AI systems in production, focusing on creating evaluation frameworks (golden datasets, LLM-as-judge pipelines) to ensure agent reliability and trustworthiness.
Role type
Senior AI Engineer (Agentic Systems & Evaluation)
Builds
Agentic systems, evaluation frameworks, golden datasets, and CI/CD integration for agent regression testing.
Domain
Artificial Intelligence / Agentic Systems / LLM Evaluation
Deliverable
production ML models
Required skills
Agent graph design (LangGraph), LLM evaluation pipeline design, Python (FastAPI, PostgreSQL), LLM API integration (Anthropic, OpenAI, Azure, Bedrock, Vertex AI), Tracing tools (Langfuse, LangSmith), Reliability engineering (retries, error handling)
Preferred skills
Experience with supervisor/sub-agent patterns, Statistical reasoning for evaluation (CIs, sample sizes)
Technologies
LangGraph, FastAPI, PostgreSQL, Langfuse, LangSmith, Anthropic, OpenAI, Azure Foundry, Bedrock, Vertex AI
Responsibilities
Build and extend agentic systems including new tools and sub-agents; Develop and maintain evaluation frameworks with golden datasets and LLM-as-judge pipelines; Integrate evaluation regressions into CI/CD to block PRs; Collaborate with R&D to automate engineering work based on customer insights.
Seniority
Senior, hands-on IC