SWE-bench Verified
codingSWE-bench Verified tests whether AI can resolve real GitHub issues from popular open-source Python repositories. "Verified" means human annotators confirmed each task is solvable and has correct test cases.
View paper / source5
Models Tested
80.9
Best Score
79.0
Average Score
0–100
Scale Range
1.5x
Weight
How It Works
The model receives a GitHub issue description and the repository state. It must generate a code patch that resolves the issue. Success is measured by whether the generated patch passes the repository's test suite.
Why It Matters
SWE-bench is the gold standard for real-world coding ability. Unlike HumanEval's isolated functions, SWE-bench requires understanding large codebases, diagnosing bugs, and writing patches that work within existing architecture.
Limitations
Only tests Python repositories. Setup and execution is computationally expensive. Some issues require domain-specific knowledge beyond pure coding ability.
Leaderboard — SWE-bench Verified
| # | Model | Provider | Score | |
|---|---|---|---|---|
| 🥇 | Claude Opus 4.5 (batch) | Anthropic | 80.9 | |
| 🥈 | Claude Opus 4.6 (batch) | Anthropic | 80.8 | |
| 🥉 | GPT-5.2 (batch) | OpenAI | 80.0 | |
| 4 | Claude Sonnet 4.5 (batch) | Anthropic | 77.2 | |
| 5 | Gemini 3 Pro | 76.2 | |