AI Evaluation Subject Matter Expert
Core
Develop evaluation frameworks, test methods, and metrics to assess AI, ML, and automation capabilities for the Navy's Next Generation CANES environment, ensuring they are mission-relevant, secure, and reliable in tactical, disconnected, and limited-bandwidth conditions.
Role type
Senior AI Evaluation Subject Matter Expert (IC)
Builds
Evaluation frameworks, test plans, datasets, risk assessments, and technical reports for AI-enabled network operations and cyber defense systems.
Domain
Defense / Cybersecurity / AI Evaluation
Deliverable
production ML models | product features | dashboards & analysis | research | client delivery
Required skills
AI/ML evaluation methods, test design, performance metrics, validation approaches, operational risk assessment, cybersecurity fundamentals, data analysis, technical reporting, cross-functional collaboration
Preferred skills
Navy/DoD mission system experience, CANES/ADNS/NAVWAR familiarity, generative AI/LLM evaluation, responsible AI governance, adversarial ML testing, MLOps/model monitoring
Technologies
Generative AI, Large Language Models, Retrieval-Augmented Generation, Autonomous Agents, AI-assisted cyber tools, Predictive Analytics, MLOps, DevSecOps
Responsibilities
Develop AI evaluation strategies and test plans; Assess AI capabilities for operational relevance and cyber risk; Design test scenarios reflecting Navy afloat operating conditions; Define data requirements and ground truth methods; Assess AI model performance using metrics like accuracy, latency, and robustness; Evaluate risks including hallucination, bias, and adversarial manipulation; Support AI red teaming and safety reviews; Produce technical reports and executive summaries for leadership.
Seniority
Senior, hands-on IC
