Senior Software Engineer – Open Source & SWE-Bench Evaluation
Core
Evaluate real-world software engineering tasks derived from GitHub issues and pull requests for technical soundness, reproducibility, and test quality.
Role type
Senior Software Engineer (Open Source & Benchmark Evaluation)
Builds
High-quality coding benchmarks and evaluation datasets for AI/ML research
Domain
Software Engineering, Open Source, AI/ML Evaluation
Deliverable
production ML models
Required skills
Reviewing pull requests, debugging complex codebases, unit testing, dependency management, Git workflows, technical writing
Preferred skills
Open-source contributions, SWE-Bench experience, Python ecosystem (Django, Flask, pytest), Docker/CI/CD, test isolation, benchmark design
Technologies
GitHub, Git, Python, Docker, CI/CD, pytest, conda, pip
Responsibilities
Review coding tasks from real GitHub issues, assess problem statement clarity, evaluate unit test correctness and coverage, identify flaky tests and environment issues, determine task reproducibility, assess task difficulty, provide acceptance recommendations
Seniority
Senior, hands-on IC
