Senior Software Development Engineer in Test (SDET), QA - AI Systems
Core
Senior SDET owning quality for autonomous AI agent systems in heavy industries, combining deep test automation with hands-on manual/exploratory testing and red-teaming.
Role type
Senior IC SDET (AI Systems)
Builds
Production-grade evaluation harnesses, simulation environments, CI/CD quality gates, and test frameworks for LLM-based agents.
Domain
AI/ML Agents, Heavy Industry, Autonomous Systems
Deliverable
production ML models
Required skills
Test automation, Python, CI/CD, LLM agent testing, Red-teaming, Test data management, Exploratory testing, API/Integration testing, Quality metrics definition, Mentorship
Preferred skills
Human-in-the-loop evaluation, Performance/load testing, ML/Data Science background, Observability tools, Industrial protocols (OPC UA, MQTT), UAT support
Technologies
Python, GitHub Actions, GitLab CI, Jenkins, Docker, Kubernetes, promptfoo, DeepEval, RAGAS, LangSmith, k6, Locust, Langfuse, OpenTelemetry, Grafana, Datadog
Responsibilities
Design and maintain test frameworks and evaluation harnesses for LLM-based agents; Build automated pipelines integrating evals into CI/CD; Perform hands-on exploratory testing and manual red-teaming of agent behavior; Create and manage golden datasets and labeled test sets; Define quality metrics and dashboards for agent performance; Mentor engineers on testing strategies for probabilistic systems.
Seniority
Senior, hands-on IC