Software Engineer L5 - AI Observability & Agent Evaluation
Core
Build frameworks, SDKs, and platforms for AI/ML observability, model evaluation, and vendor integration to make AI systems transparent and production-ready.
Role type
Senior IC platform engineer (AI observability & evaluation)
Builds
Reusable observability infrastructure, evaluation primitives, SDKs, connectors, and dashboards for ML/GenAI teams
Domain
AI/ML infrastructure, GenAI, LLM evaluation, distributed systems
Deliverable
production ML models | infrastructure
Required skills
Python, Java/Go/Scala, production observability, ML/LLM evaluation, distributed systems, cloud platforms, vendor integration
Preferred skills
ML/LLM observability tools, LLM/GenAI application development, agentic systems evaluation
Technologies
Prometheus, Grafana, Datadog, OpenTelemetry, ELK, OpenSearch, Jaeger, Tempo, AWS, GCP, Azure
Responsibilities
Build observability frameworks for metrics, logs, and traces across inference and batch; Create primitives for monitoring model performance, data quality, and drift; Develop evaluation frameworks for LLMs and agentic systems; Lead build-vs-buy evaluations and integrate vendor tooling; Build reusable libraries and templates for observability-by-default; Provide dashboarding, alerting, and SLO/SLI building blocks
Seniority
Senior, hands-on IC