Analyste Benchmarking et Red Teaming des systèmes d'IA générative - Risques - La Defense, Ile de France, France
Core
Conduct benchmarking and Red Teaming to evaluate the limits, vulnerabilities, and failure scenarios of generative AI models, specifically focusing on agent systems.
Role type
Junior IC model risk analyst (generative AI)
Builds
Internal risk validation tools and Red Teaming applications for LLMs
Domain
Banking / Generative AI / Model Risk Management
Deliverable
production ML models
Required skills
Machine learning fundamentals, Python, Transformers, LangChain/LangGraph, Git, API deployment, containerization, cloud deployment
Preferred skills
Literature review, adversarial testing design, comparative analysis, documentation, workshop facilitation
Technologies
Python, Transformers, LangChain, LangGraph, Git, APIs, Containers, Cloud
Responsibilities
Review literature on LLM benchmarking and agent systems, deploy evaluation methods in internal tools, develop Red Teaming application features, design and execute adversarial test campaigns, compare testing strategies, document findings and present to the team
