Senior AI Engineer
Core
Build and run evaluation systems (production tracing, LLM judges, goldsets) to measure and improve the quality of AI agents for the construction industry.
Role type
Senior AI Engineer (Evaluation & Observability)
Builds
Production tracing infrastructure, online LLM judge systems, data pipelines for goldsets, and agentic feedback loops.
Domain
AI Engineering / Construction Industry
Deliverable
production ML models
Required skills
Python, LLM evaluation (LLM-as-judge, rubric design), LLM observability tools (Langfuse), data pipeline/ETL, data warehousing (Databricks/Snowflake), system architecture
Preferred skills
TypeScript, mentoring
Technologies
Langfuse, Databricks, Python, TypeScript
Responsibilities
Improve online evals (LLM judge accuracy, calibration), build production trace infrastructure, maintain data pipelines for goldsets, build agentic loops to mine failures, optimize runtime/cost/reliability of eval pipelines, review designs and mentor engineers
Seniority
Senior, hands-on IC with architecture and mentorship