AI Evaluation Engineer
Core
Design and maintain enterprise data environments and benchmarks to evaluate AI analysts' performance on complex, production-scale data tasks.
Role type
Senior IC AI evaluation engineer (data engineering focus)
Builds
Realistic multi-system enterprise data environments, benchmark datasets, and evaluation harnesses for AI analysts
Domain
Data engineering + AI evaluation
Deliverable
production ML models
Required skills
Advanced SQL, Data modeling, Python, Data engineering, LLM evaluation, Benchmark design, Synthetic data generation, Multi-system environment setup, Regression tracking
Preferred skills
Healthcare/Finance/Product Analytics domain knowledge, LLM evaluation frameworks, Streaming/data-lake stacks
Technologies
Snowflake, BigQuery, Redshift, Postgres, pandas, NumPy, Polars, S3, Parquet, dbt, Airflow, Spark, DuckDB
Responsibilities
Design and stand up multi-system enterprise environments; Engineer calibrated benchmark datasets; Curate task libraries and golden SQL; Own the evaluation harness and scoring integrity
Seniority
Senior, hands-on IC

