Member of Technical Staff (Quality Engineering)
Core
Building evaluation systems, test infrastructure, and compliance checks for an AI platform that generates regulated pharmaceutical marketing content.
Role type
Senior Quality Engineering Lead (AI/ML)
Builds
Evaluation frameworks, CI/CD quality gates, and end-to-end test suites for probabilistic AI models.
Domain
Life Sciences / AI / Pharmaceutical Marketing
Deliverable
production ML models
Required skills
Python, Playwright, manual/exploratory QA, non-deterministic ML/LLM testing, evaluation methods (golden datasets, LLM-as-judge), statistical quality thinking, error analysis, independent ownership, clear communication.
Preferred skills
Agent/LLM frameworks (LangChain, LangGraph), eval/observability tools (LangSmith, Langfuse, RAGAS), TypeScript/React, adversarial/red-team testing, async Python services, MLOps/LLMOps.
Technologies
Python, Playwright, LangChain, LangGraph, LangSmith, Langfuse, RAGAS, Promptfoo, OpenAI Evals, TypeScript, React.
Responsibilities
Design evaluation systems to score AI output quality; ensure model/prompt changes are safe; test agent failures and compliance paths; own end-to-end testing with Playwright; run manual/exploratory QA; build CI/CD quality gates; use production for drift detection; harden background jobs; set testing standards.
Seniority
Senior, hands-on IC