Senior AI Engineer - LLM Evaluation
Core
Establish evaluation standards, build automated validation frameworks, and drive observability practices for Generative AI systems while contributing to the design and development of AI-powered applications.
Role type
Senior AI Engineer (LLM Evaluation & Quality Engineering)
Builds
Cloud-native AI solutions, RAG systems, agentic workflows, and LLM-based solutions for the Architecture, Engineering, and Construction (AEC) industry.
Domain
Generative AI, Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), Agentic Workflows, AEC
Deliverable
production ML models
Required skills
LLM evaluation, automated testing solutions, model validation pipelines, Python, cloud-native application deployment, observability, CI/CD integration, RAG pipeline construction, MCP-based integrations
Preferred skills
AI evaluation frameworks (LangSmith, DeepEval, TruLens, Promptfoo, OpenAI Evals), agentic AI frameworks (LangGraph, CrewAI, MCP, A2A), deep learning architectures (Transformers, PyTorch, Lightning, Ray), enterprise knowledge systems, semantic/vector search
Technologies
Python, AWS, Microsoft Azure, Google Cloud Platform (GCP), LangSmith, DeepEval, TruLens, Promptfoo, OpenAI Evals, LangGraph, CrewAI, PyTorch, Lightning, Ray, Model Context Protocol (MCP)
Responsibilities
Lead AI evaluation and quality strategy for Generative AI and ML-powered systems; Design and implement scalable evaluation frameworks for LLMs, RAG systems, and agentic workflows; Fine-tune, evaluate, benchmark, and deploy LLMs in production; Implement evaluation and observability standards including regression testing and monitoring; Define and track AI quality metrics (accuracy, robustness, bias, drift, grounding, latency, safety, hallucination); Build automated evaluation pipelines integrated into CI/CD workflows; Build scalable RAG pipelines using enterprise data sources; Integrate AI agents with Autodesk platforms using MCP-based integrations; Contribute to reusable AI platform capabilities, SDKs, and orchestration patterns.
Seniority
Senior, hands-on IC