Principal Software Engineer for Copilot Evals
Core
Own and drive the end-to-end CADET architecture for the customer-signal and evaluation system, creating scalable services to reproduce customer pain points and automate evaluation execution.
Role type
Principal Software Engineer (AI Evaluation Infrastructure)
Builds
Scalable services and pipelines for automated grounding-data generation, intent classification, deduplication, clustering, taxonomy assignment, coverage analysis, loss-pattern detection, and work-item routing.
Domain
AI/ML Infrastructure, Large Language Model (LLM) Evaluation, Agent Systems
Deliverable
production ML models | infrastructure
Required skills
Designing large-scale distributed systems, LLM technologies, agent systems, AI/ML platforms, retrieval, tool-using agents, prompt and rubric management, human-in-the-loop systems, automated diagnosis, data and workflow platforms (batch/streaming/unstructured), RLEs, synthetic environments, post-training pipelines, large-scale evaluation execution, integrating customer feedback and telemetry.
Preferred skills
None explicitly stated as preferred; all listed capabilities are required.
Technologies
C, C++, C#, Java, JavaScript, Python
Responsibilities
Resolve architectural ambiguity and create reusable capabilities across partner teams, establish data contracts and extensible schemas for customer signals and evaluation assets, develop quality-attribution workflows combining automated diagnosis with human review, create observability and benchmarks for pipeline health and agent behavior, design safe human-in-the-loop gates for ambiguous data and sensitive evidence, mentor engineers and raise engineering quality through clear interfaces and testing.
Seniority
Principal, hands-on IC with organizational leadership