Software Engineering Manager, AI Platform
Core
Build infrastructure and tooling for measuring, benchmarking, and improving the quality of AI agents (Search, RFI Create, Invoice) and future agentic products.
Role type
Senior IC Engineering Manager (AI Evaluation Platform)
Builds
Evaluation infrastructure, quality measurement pipelines, and developer tooling for AI agents.
Domain
Construction-tech / Enterprise B2B SaaS / AI Agents
Deliverable
production ML models
Required skills
Team leadership, technical vision & roadmap definition, evaluation methodologies (precision/recall, LLM-as-judge, A/B testing), backend systems & data pipelines, distributed infrastructure, search relevance systems, construction-tech domain knowledge
Preferred skills
Experience with RAG pipelines, multi-step agents, tool-use agents, evaluation frameworks (RAGAS, DeepEval, LangFuse), human-in-the-loop workflows
Technologies
Python, Go
Responsibilities
Lead and grow a team of engineers focused on evaluation infrastructure; Define technical vision and roadmap for offline and online evaluations; Partner with AI/ML and Product teams to define quality metrics; Design and deliver user-facing evaluation tools; Build frameworks for human-in-the-loop evaluation; Establish CI/CD quality gates; Drive engineering excellence and mentor engineers
Seniority
Senior, hands-on IC