CareerPlan

Member of Technical Staff (Quality Engineering)

New York CityFull-time$160k–$300k2026-06-26 → 2026-11-19
On-siteVisa sponsorship offered

Core

Building evaluation systems, test infrastructure, and compliance checks for an AI platform that generates regulated pharmaceutical marketing content.

Role type

Senior Quality Engineering Lead (AI/ML)

Builds

Evaluation frameworks, CI/CD quality gates, and end-to-end test suites for probabilistic AI models.

Domain

Life Sciences / AI / Pharmaceutical Marketing

Deliverable

production ML models

Required skills

Python, Playwright, manual/exploratory QA, non-deterministic ML/LLM testing, evaluation methods (golden datasets, LLM-as-judge), statistical quality thinking, error analysis, independent ownership, clear communication.

Preferred skills

Agent/LLM frameworks (LangChain, LangGraph), eval/observability tools (LangSmith, Langfuse, RAGAS), TypeScript/React, adversarial/red-team testing, async Python services, MLOps/LLMOps.

Technologies

Python, Playwright, LangChain, LangGraph, LangSmith, Langfuse, RAGAS, Promptfoo, OpenAI Evals, TypeScript, React.

Responsibilities

Design evaluation systems to score AI output quality; ensure model/prompt changes are safe; test agent failures and compliance paths; own end-to-end testing with Playwright; run manual/exploratory QA; build CI/CD quality gates; use production for drift detection; harden background jobs; set testing standards.

Seniority

Senior, hands-on IC