大模型智能标注与评测算法工程师（J106467）
Core
Design, develop, and deploy an intelligent annotation and evaluation platform for large language models (LLMs), building data production systems and evaluation agents to support model selection, traffic allocation, and business decisions.
Role type
Senior IC machine-learning engineer (LLM evaluation & annotation platform)
Builds
Automatic annotation systems, Judge/Reward models, evaluation agents, and automated iteration loops for LLMs
Domain
Artificial Intelligence / Large Language Models / Data Engineering
Deliverable
production ML models
Required skills
Python, PyTorch, LLM evaluation methods, Judge/Reward model development, data engineering, annotation quality governance, statistical calibration, bias detection
Preferred skills
Healthcare domain experience, Agent development, RLHF, large-scale model evaluation platform experience
Technologies
Python, PyTorch
Responsibilities
Design and develop the overall architecture of the intelligent annotation and evaluation platform; Build data production systems combining automatic, weakly-supervised, and human-in-the-loop annotation; Develop Judge Models, Reward Models, and evaluation Agents for various evaluation metrics; Establish evaluation calibration mechanisms using gold standard sets and expert reviews; Build online feedback-driven loops for bad case discovery, data generation, and model iteration; Identify and mitigate model biases to improve system stability and interpretability; Support model selection and traffic allocation decisions based on evaluation results.