Senior AI Engineer (m/f/n)
Core
Building and operating the Evaluations Platform for Von Halsky, InPost's conversational AI shopping assistant, to gate releases and measure LLM performance via automated judging pipelines and eval datasets.
Role type
Senior AI Engineer (Technical Lead)
Builds
LLM-judge pipelines, eval datasets, golden sets, and root-cause analysis tools for a production conversational AI system.
Domain
E-commerce / Conversational AI / LLM Evaluation
Deliverable
production ML models
Required skills
LLM-based system production experience, LLM evaluation (LLM-as-judge, calibration, regression suites), prompt and context engineering, agentic primitives, SQL, cloud infrastructure, Python, AI-assisted development tooling
Preferred skills
Harness and loop engineering, auto-improving systems, adversarial robustness and red-teaming, e-commerce domain experience
Technologies
Python, SQL, LangFuse, Braintrust, Weave, Claude Code, Cursor, Copilot
Responsibilities
Calibrate LLM-judge pipelines against human labels, manage eval datasets and golden sets with coverage across intents, perform root-cause analysis on real conversations, detect abusive or anomalous sessions, define measurable quality definitions for stakeholders, mentor engineers and scope technical work
Seniority
Senior, hands-on IC with technical lead responsibilities
