Software Engineer, Health Evaluations
Core
Define end-to-end evaluation strategy for LLM agents deployed in a Mayo Clinic partnership to ensure patient safety, factual accuracy, and high-quality interactions.
Role type
Senior IC Evals Engineer (Healthcare AI)
Builds
Layered evaluation systems (static benchmarks, adaptive simulations, agentic evaluations), automated graders, and dashboards for clinical AI performance.
Domain
Healthcare / Clinical AI / LLM Evaluation
Deliverable
production ML models
Required skills
LLM evaluation strategy, LLM-as-judge design, user simulation, agentic/trajectory evaluation, Python, data pipeline development, clinical workflow understanding, HIPAA/PHI handling, cross-functional project leadership.
Preferred skills
Healthcare/health tech domain experience, FDA/EU AI Act regulatory familiarity, experience building tooling for LLM products.
Technologies
Python, LLM frameworks, evaluation dashboards
Responsibilities
Design clinically-grounded evaluation guidelines and rubrics; build automated evaluations and calibrate autograders against expert clinician annotation; evaluate patient-facing AI for safety and factuality; partner with Mayo clinicians to run evaluations; translate evaluation insights into product improvements; monitor model performance trends and clinical-AI benchmarks.
Seniority
Senior, hands-on IC
