Data Scientist
Core
Designing, implementing, and maintaining ML model validation frameworks, regression test suites, and production monitoring to ensure reliability and accuracy of ML and LLM systems.
Role type
Senior IC Data Scientist (ML Model Validation & Monitoring)
Builds
Production ML and LLM models, evaluation frameworks, and monitoring pipelines
Domain
SaaS / Small Business Management / Machine Learning
Deliverable
production ML models
Required skills
Python, Pandas, Scikit-Learn, XGBoost, PyTorch or TensorFlow, SQL, statistical concepts (bias-variance, calibration, A/B testing, data drift), LLM evaluation frameworks (RAGAS, Eleuther AI Eval Harness), custom metric design, model architecture reasoning
Preferred skills
Snowflake, Snowpark, CI/CD regression testing, prompt engineering, SaaS environment experience
Technologies
Python, Pandas, Scikit-Learn, XGBoost, PyTorch, TensorFlow, SQL, RAGAS, Eleuther AI Eval Harness, Snowflake, Snowpark
Responsibilities
Design and maintain ML model validation frameworks with custom metrics and statistical tests; Build and own regression test suites for ML and LLM models; Develop and execute MCP evaluations for model capabilities and failure modes; Monitor models in production using statistical process control and drift detection; Collaborate on ML model architecture design and refinement; Document evaluation methodologies and monitoring runbooks
Seniority
Senior, hands-on IC
