Senior AI Systems Quality Engineer
Core
Engineering quality, reliability, and trust into production-grade AI systems for mission-critical healthcare environments by designing automated testing frameworks and evaluation pipelines.
Role type
Senior IC AI Systems Quality Engineer
Builds
Production-grade automated validation frameworks, test harnesses, evaluation pipelines, and an AI testing platform integrated with Databricks and MLflow.
Domain
Healthcare + AI/LLM Systems
Deliverable
production ML models
Required skills
Python, TypeScript, CI/CD integration, AWS cloud-native architectures, automated testing framework design, LLM/agentic workflow validation, non-deterministic output evaluation, drift detection, bias and fairness testing, release-readiness criteria definition, system contract and guardrail design.
Preferred skills
Databricks, Medallion architecture, MLflow, observability tools (Datadog, Prometheus, Grafana), LLM evaluation techniques, prompt and agent behavior versioning, adversarial test scenario generation.
Technologies
Databricks, MLflow, AWS, Python, TypeScript, Datadog, Prometheus, Grafana
Responsibilities
Build and deploy production-grade automated validation frameworks and evaluation pipelines across the AI development lifecycle; Design and evolve an AI testing platform integrated with Databricks and MLflow for traceability and auditability; Create large-scale, scenario-based test suites covering edge cases and system failure modes; Validate agentic orchestration behaviors including tool usage, memory, and decision logic; Embed quality-by-design principles by defining system contracts, guardrails, and safe-degradation patterns; Define measurable quality signals for LLM systems including grounding, hallucination rates, and latency; Integrate automated quality gates into CI/CD pipelines for continuous validation; Establish measurable release-readiness criteria and support go/no-go decisions based on quality thresholds.
Seniority
Senior, hands-on IC
