Applied Scientist, Customer Experience and Business Trends, Customer Experience and Business Trends
Core
Designing scientific methodologies to evaluate autonomous coding agents and their experiences on AWS, focusing on measuring agent behavior and decision-making rather than self-reported outcomes.
Role type
Research scientist specializing in GenAI and agent evaluation
Builds
Scientific measurement frameworks and evaluation metrics for autonomous agents
Domain
Cloud computing, Generative AI, Agent-based systems
Deliverable
production ML models
Required skills
Deep learning for NLP, agentic/tool-using systems, cloud architecture knowledge, experimental design, causal inference, model validation
Preferred skills
AWS services expertise, big data processing (Spark, Sagemaker), prior research scientist experience
Technologies
Python, Java, C++, AWS Redshift, EMR, Spark, Sagemaker
Responsibilities
Develop novel methodologies for evaluating autonomous agents, design rigorous experiments for causal insight, apply cloud architecture knowledge to agent decisions, create verification methods for evidence-grounded conclusions, design evaluation metrics for agent outcomes, publish research at scientific venues
Seniority
Senior, hands-on IC