Data Scientist, Agent
Core
You own how Lovable's AI agent is measured and improved by building evaluation systems, experiments, and telemetry analysis to raise success rates and cut errors.
Role type
Senior IC data scientist (LLM evaluation & agent observability)
Builds
Eval systems, experiment frameworks, and continuous monitoring tooling for an AI agent
Domain
AI agents, LLM evaluation, observability, software creation
Deliverable
production ML models
Required skills
LLM evaluation, observability, SQL, Python, applied statistics, A/B test design, telemetry analysis
Preferred skills
Noisy outcome analysis, agent behavior modeling, entrepreneurial mindset
Technologies
Braintrust, OTEL tracing, BigQuery, PubSub, Hex, GCP
Responsibilities
Define and own agent quality metrics (success, completion, error rates); Build eval systems and experiment frameworks to validate agent changes; Convert agent traces and telemetry into concrete fixes; Build tooling for continuous evaluation; Set standards for judging agent behavior without answer keys
Seniority
Senior, hands-on IC
