Principal AI Architect - M365 IC3 Team (Intelligent Conversation and Communications Cloud)
Core
Define technical vision and architecture for AI, LLM, RAG, and agent evaluation systems; design evaluation frameworks to assess product quality across the lifecycle from pre-implementation to post-ship.
Role type
Principal AI Architect (Evaluation & Architecture)
Builds
Evaluation frameworks, dashboards, scorecards, quality reports, product-health views, and systems connecting telemetry, offline evals, human judgment, and automated evals.
Domain
Enterprise AI, LLMs, RAG, Agentic Systems, Product Quality
Deliverable
production ML models | product features | dashboards & analysis
Required skills
LLM evaluation design, RAG evaluation, agentic system behavior analysis, architecture of complex multi-component systems, CI/CD integration for evals, statistical analysis of nondeterministic systems, technical strategy definition, cross-functional leadership, coding and technical design, product lifecycle management
Preferred skills
Experience with enterprise search, grounding, indexing, ranking, permissions, and relevance systems; experience with red teaming, synthetic data, model monitoring, and responsible AI practices
Technologies
LLMs, RAG, agentic systems, ML pipelines, experimentation systems, CI/CD pipelines, data visualization systems
Responsibilities
Design evaluation frameworks for pre-implementation, development, launch, and post-ship phases; identify behavioral gaps, failure modes, and model limitations; partner with engineering and data science teams to integrate ML/DS methods into product codebases; integrate evals into build pipelines and release gates; mentor senior engineers on building reliable evaluation infrastructure; drive architecture decisions across components, services, and data pipelines; work with product leaders to prioritize evaluation investments.
Seniority
Principal, strategy & mentorship
