CareerPlan

Software Engineer, Health Evaluations

New YorkFull-time$143k–$275k2026-11-17 → 2026-11-19
Hybrid

Core

Define end-to-end evaluation strategy for LLM agents deployed in a Mayo Clinic partnership to ensure patient safety, factual accuracy, and high-quality interactions.

Role type

Senior IC Evals Engineer (Healthcare AI)

Builds

Layered evaluation systems (static benchmarks, adaptive simulations, agentic evaluations), automated graders, and dashboards for clinical AI performance.

Domain

Healthcare / Clinical AI / LLM Evaluation

Deliverable

production ML models

Required skills

LLM evaluation strategy, LLM-as-judge design, user simulation, agentic/trajectory evaluation, Python, data pipeline development, clinical workflow understanding, HIPAA/PHI handling, cross-functional project leadership.

Preferred skills

Healthcare/health tech domain experience, FDA/EU AI Act regulatory familiarity, experience building tooling for LLM products.

Technologies

Python, LLM frameworks, evaluation dashboards

Responsibilities

Design clinically-grounded evaluation guidelines and rubrics; build automated evaluations and calibrate autograders against expert clinician annotation; evaluate patient-facing AI for safety and factuality; partner with Mayo clinicians to run evaluations; translate evaluation insights into product improvements; monitor model performance trends and clinical-AI benchmarks.

Seniority

Senior, hands-on IC