QA / Evaluation Lead
Core
Design and own the evaluation framework and quality assurance methodology for a federal AI data services platform, ensuring data outputs meet rigorous trust and safety standards.
Role type
Senior IC QA/Evaluation Lead (AI/ML Data Quality)
Builds
A governed data services platform with synthetic data integration and Databricks write-back capabilities for federal critical infrastructure.
Domain
Federal government / AI/ML data engineering / Trust & Safety
Deliverable
production ML models | dashboards & analysis
Required skills
Inter-annotator agreement (IAA) methodology (Cohen's kappa, Fleiss' kappa, Krippendorff's alpha), evaluation framework design, sampling-based quality control, confidence-threshold escalation routing, Python for statistical analysis, drift detection
Preferred skills
DoD or IC data quality program experience, CVAT annotation platform configuration, FMV/video annotation quality standards
Technologies
Databricks, Python, CVAT
Responsibilities
Design IAA methodology for demonstration corpus, define evaluation framework architecture including drift detection gates, configure sampling-based QC across annotation paths, implement confidence-threshold escalation routing, validate quality scoring and IAA computation, support AI Solutions Engineer on model API validation, produce evaluation framework documentation
Seniority
Senior, hands-on IC
