Senior Software Engineer – Open Source & SWE-Bench Evaluation
Core
Evaluating real-world software engineering tasks derived from open-source repositories for technical soundness, reproducibility, and quality.
Role type
Senior IC software engineer (open-source evaluation)
Builds
High-quality coding benchmarks and evaluation datasets for AI/ML research
Domain
Software engineering, open-source development, AI evaluation
Deliverable
production ML models
Required skills
reviewing pull requests, debugging complex codebases, unit testing, dependency management, Git workflows, technical writing, environment configuration
Preferred skills
open-source contributions, SWE-Bench experience, Python frameworks (Django, Flask, pytest), Docker/CI/CD, test isolation, RLHF/benchmark development
Technologies
GitHub, Git, Python, Docker, CI/CD, pytest, conda, pip
Responsibilities
Reviewing coding tasks from GitHub issues/PRs, assessing problem statement clarity, evaluating unit test correctness and coverage, identifying flaky tests and environment issues, determining task reproducibility, assessing task difficulty, providing acceptance recommendations
