Principal AI Tool Engineer for Software Automation/Testing
Core
Design and develop testing frameworks, automated pipelines, and evaluation tools for AI/ML models and LLM applications to ensure accuracy, robustness, and safety.
Role type
Principal AI Tool Engineer (Software Automation/Testing)
Builds
Automated testing frameworks, validation pipelines, benchmarking systems, and alerting infrastructure for AI models.
Domain
Cybersecurity, Artificial Intelligence, Machine Learning, Large Language Models (LLMs)
Deliverable
production ML models | product features
Required skills
Python, testing frameworks (PyTest, unittest), machine learning concepts, API integration, distributed systems, CI/CD pipelines, model evaluation techniques, observability tools
Preferred skills
LLMs and prompt engineering, evaluation tools (LangChain, OpenAI Evals), AI safety and bias detection, cloud platforms (AWS, GCP, Azure), synthetic data generation
Technologies
Python, PyTest, unittest, LangChain, OpenAI Evals, AWS, GCP, Azure
Responsibilities
Design testing frameworks for AI/ML models; Build automated pipelines for model validation and regression testing; Create evaluation datasets and test scenarios; Implement metrics for accuracy, robustness, and latency; Develop tools for prompt testing and hallucination detection; Monitor model performance in production; Debug model behavior and identify root causes of failures.
Seniority
Principal, hands-on IC with strategy & mentorship