Evaluation Engineer, Applied AI
Core
Build frontier LLM evaluation benchmarks and frameworks to assess model capabilities in real-world industry use cases (e.g., CAD, finance) and guide product development.
Role type
Senior IC evaluation engineer (LLM)
Builds
Industry-specific LLM benchmarks, evaluation agents, and data pipelines
Domain
Artificial Intelligence / Large Language Models / Enterprise Software
Deliverable
production ML models
Required skills
Python, LLM training and inference, benchmark design, data sourcing and cleaning, stakeholder workshops, technical writing
Preferred skills
AI evaluation experience, synthetic data generation, open-source contributions, domain expertise (engineering/finance/science)
Technologies
Python, LLM frameworks
Responsibilities
Run discovery workshops with customers and experts to define use cases; Design and build benchmarks where agents operate inside real software; Stress-test existing evals for flaws; Source data or build synthetic proxies; Present results to senior leaders; Help teams build reusable evaluation tools
Seniority
Senior, hands-on IC
