Data Scientist
Core
Design evaluation harnesses, build LLM-as-judge pipelines, and develop production predictive models for an AIOps platform using small, locally-hosted language models.
Role type
Senior IC Data Scientist (LLM Evaluation & Predictive Modeling)
Builds
AIOps platform with self-healing, self-optimizing capabilities for enterprise IT operations
Domain
IT Operations (AIOps) + Large Language Models (LLM)
Deliverable
production ML models
Required skills
Applied statistics, Python, SQL, LLM evaluation design, predictive modeling, time-series forecasting, anomaly detection, adversarial testing, data visualization
Preferred skills
Small/local model optimization, retrieval-augmented generation (RAG) evaluation, agentic framework analysis, red-teaming, IT operations domain knowledge, big data stores
Technologies
Python, SQL, LLM evaluation frameworks, time-series libraries, cloud platforms
Responsibilities
Design and own evaluation harnesses for LLM and agentic outputs; Build and calibrate LLM-as-judge pipelines; Build production models for forecasting, anomaly detection, and trend analysis; Red-team the system for prompt injection and edge-case discovery; Monitor output-distribution and quality drift in production; Quantify system economics and connect to customer-facing value metrics
Seniority
Senior, hands-on IC
