Scale AI's SWE-Bench Pro

AI agents for software engineering are rapidly advancing, but are benchmarks keeping up? With frontier models scoring so highly on [SWE-Bench Verified](https://openai.com/index/introducing-swe-bench-verified/), we wanted to raise the bar and develop a more realistic, contamination-resistant, human-augmented benchmark. [SWE-Bench Pro](https://scale.com/research/swe_bench_pro) picks up where SWE-Bench Verified leaves off, with more diverse tasks, increased difficulty, and code that models have not yet seen. On SWE-Bench Pro, the same four frontier models lead the pack, but at considerably lower scores.

## A Benchmark to Meet Today’s Coding Needs

SWE-Bench Pro was designed to accurately measure the ability of coding agents to meet the needs of today. It contains: **1,865 total instances**(731 public, 858 held-out, and 276 commercial) across **41 repositories** (11 public, 12 held-out, and 18 from enterprise startups).

SWE-Bench Pro solves several key challenges in evaluating AI coding agents:

1. Data Contamination
   - The Problem: Many benchmarks use code that models have likely seen during training. This makes it difficult to know if a model is genuinely solving a problem or just recalling a memorized solution.

- The Solution: We use code that models haven't been trained on. This is sourced from public codebases governed by strong copyleft licenses (e.g., GPL), whose "viral" nature and legal complexities make them highly likely to be excluded from training data, and completely private, commercial codebases from Scale's internal assets.

1. Limited Task Diversity
   - The Problem: Current benchmarks fail to capture the full spectrum of real-world software engineering challenges, often focusing on simple utility libraries.

- The Solution: We source tasks from a diverse portfolio of complex repositories, including consumer-facing applications, B2B services, and developer tools. Each repository contributes 50-100 tasks to ensure models must genuinely understand the code, not just overfit to a single project's style.

1. Oversimplified Problems & Unrealistic Difficulty
   - The Problem: Previous benchmarks tend to filter out ambiguous or underspecified issues, which doesn't reflect a real developer's workflow.

- The Solution: We preserve these challenging tasks. Because a developer's original commit messages are often unstructured or incomplete, we use a human-augmented process to enhance them. Human experts produce a clear problem statement and a list of requirements that specify the expected behavior but not how to implement the solution, preserving the core technical challenge. These tasks require substantial changes, averaging 107.4 lines of code across 4.1 files.

1. Unreliable and Irreproducible Testing
   - The Problem: Without a consistent setup, it's hard to know if a solution works or if the environment is just configured incorrectly.

- The Solution:

## Results: SWE-Bench Verified vs. SWE-Bench Pro

We ran frontier models on Pro usin

... (truncated, 8 KB total)