Project Lion - Prompt Engineer - United States (Remote, Part-Time)
Core
End-to-end technical migration of templates to LLM autoraters using prompt engineering to maximize model performance.
Role type
Prompt Engineer
Builds
LLM autorater systems for template evaluation
Domain
AI / Machine Learning / NLP
Deliverable
production ML models
Required skills
Prompt Engineering, Large Language Model (LLM) tuning, Chain-of-thought reasoning, Few-shot learning, Complex classification, Structured output generation, Edge case handling, Automated Prompt Generation (APG) tools, Automated Prompt Optimization (APO) tools, Shadowbot monitoring, Accuracy metric calculation (F1, precision, recall)
Responsibilities
Utilize APG tools to create baseline prompts for complex template clusters; Run and supervise APO tools, reviewing outputs and flagging deadlocks; Manually draft, test, and refine prompts to navigate complex architectures and handle edge cases; Monitor shadowbot runs to ensure sufficient disagreements are registered; Run prompt versions against gold data to measure autorater quality and calculate accuracy metrics; Draft technical launch readiness justifications.
