Machine Learning Engineering Technical Leader
Core
Define multi-quarter strategy for agentic AI quality, building layered evaluation systems (automated checks, LLM judges, human review) to measure, monitor, and improve AI agents in production.
Role type
Senior IC machine learning engineering technical leader (agentic AI evaluation)
Builds
Layered evaluation architecture, LLM-as-judge panels, simulation environments, and production monitoring systems for agentic AI.
Domain
Enterprise networking, observability, and agentic AI systems.
Deliverable
production ML models
Required skills
Machine learning system design, LLM evaluation and calibration, Python, deep learning frameworks (PyTorch/TensorFlow), technical leadership via influence, CI/CD integration, synthetic data generation, cost-aware inference.
Preferred skills
LLM-as-judge design, agentic system evaluation metrics, production monitoring at scale, networking/observability domain knowledge, technical community contribution.
Technologies
Python, PyTorch, TensorFlow, LLMs, CI/CD pipelines.
Responsibilities
Own evaluation roadmap from architecture to production; design layered evaluation architecture; build and calibrate multi-model LLM-as-judge panels; run human-in-the-loop flywheel; score production traffic and alert on regressions; lead simulation and adversarial testing; close the loop from failure to fix; drive common standards across teams; mentor engineers.
Seniority
Senior, hands-on IC with strategic influence
