About the job
Core
Design evaluation criteria for AI/ML services, select and operate LLMs, manage LLM Gateway/AI Agent frameworks, and optimize costs while ensuring safety and quality.
Role type
Senior LLM Operations Engineer (LLMOps)
Builds
Production LLM services, evaluation frameworks, and cost-optimized inference pipelines for Tmap Mobility's data platform.
Domain
AI/ML, LLM Operations, Data Platform
Deliverable
production ML models
Required skills
LLM evaluation design, LLM Gateway operations, AI Agent framework management, cost optimization, security risk detection, SQL, Python
Preferred skills
LLM Gateway/AI Agent policy management, LangGraph/LangChain/LlamaIndex experience, prompt/context/caching optimization, LLM governance, observability/monitoring, Databricks/AWS
Technologies
LLM Gateway, LangGraph, LangChain, LlamaIndex, Databricks, AWS
Responsibilities
Design evaluation criteria and select suitable LLMs based on quality/cost/speed/safety; lead adoption of open-source and commercial software for LLM Gateway and AI Agent frameworks; build evaluation data and standards to verify model quality; manage model registration, routing, failover, caching, and usage policies; analyze token usage and costs to optimize policies; detect AI service anomalies and security risks; establish model usage standards and access controls.
Seniority
Senior, hands-on IC
