Senior AI Systems Quality Engineer
Core
Designing and building automated validation frameworks, test harnesses, and evaluation pipelines to ensure agentic AI systems are production-ready, trustworthy, and safe for mission-critical healthcare environments.
Role type
Senior AI Systems Quality Engineer (Platform/Engineering)
Builds
Production-grade automated AI testing platforms, evaluation pipelines, and quality gates integrated into CI/CD.
Domain
Healthcare technology, Generative AI, Agentic systems
Deliverable
production ML models
Required skills
Python, TypeScript, AI testing automation, LLM/agentic workflows, CI/CD integration, AWS cloud-native architectures, non-deterministic system behavior management, system contracts and guardrails design, drift detection, bias/fairness testing, release readiness criteria definition.
Preferred skills
Databricks Medallion architecture, MLflow, observability tools (Datadog, Prometheus, Grafana), prompt engineering, adversarial test dataset generation.
Technologies
Python, TypeScript, Databricks, MLflow, AWS, CI/CD pipelines
Responsibilities
Build end-to-end automated AI testing integrated into the development lifecycle; Design and evolve an AI testing platform integrated with Databricks and MLflow; Create large-scale, scenario-based test suites to validate agentic workflows; Validate orchestration behavior and stress-test non-deterministic system behavior; Embed quality by design by defining system contracts, guardrails, and safe-degradation patterns; Define measurable quality signals for LLM systems and integrate them into CI/CD pipelines; Ensure AI validation runs automatically on model, prompt, and code changes; Build reusable libraries and components for consistent AI quality practices; Own aspects of AI release readiness based on measurable quality thresholds.
Seniority
Senior, hands-on IC