OpenAI says SWE-Bench Pro has a broken-task problem
OpenAI audited SWE-Bench Pro and estimates that about 30% of its public tasks are broken, changing how teams should read coding-agent benchmark claims.
OpenAI says its audit of SWE-Bench Pro found enough task-quality problems to retract its earlier recommendation that the benchmark should replace SWE-bench Verified. Confidence is confirmed for OpenAI's claim and limited for the wider benchmark impact, because the audit comes from one frontier lab and the benchmark maintainers may respond with fixes or a revised subset.
What changed
OpenAI published a July 8, 2026 audit of SWE-Bench Pro, a coding-agent benchmark designed for longer, more realistic software engineering tasks. The company says its pipeline flagged 200 broken tasks in the 731-task public split, while a human annotation campaign identified 249 broken tasks.
The practical number is OpenAI's estimate that roughly 30% of SWE-Bench Pro tasks are broken. The reported failure modes include overly strict tests, underspecified prompts, low-coverage tests, and at least one misleading prompt that points models toward behavior the hidden tests reject.
Key takeaways
- OpenAI no longer recommends SWE-Bench Pro as the default replacement for SWE-bench Verified.
- The audit targets the 731-task public split, not every possible private or commercial evaluation path.
- The finding matters because benchmark scores shape model marketing, agent-stack selection, and safety deployment decisions.
- Hidden tests that encode narrow implementation details can mark functionally valid patches as failures.
- Teams should treat coding-agent benchmark claims as prompts for due diligence, not as buying decisions.
| Benchmark signal | Best use | Current caveat |
|---|---|---|
| SWE-Bench Pro public split | Long-horizon coding-agent comparison | OpenAI estimates about 30% broken tasks in the audited public split |
| Vendor leaderboard claims | Quick shortlist building | May mix model quality, harness tuning, test quirks, and cost assumptions |
| Private internal evals | Workflow-specific purchasing decisions | Requires engineering time and representative tasks |
| Live repo trials | Migration and budget checks | Needs guardrails, review tracking, and rollback plans |
Why this is early
This is early because the audit is fresh, and the downstream benchmark ecosystem has not settled on a replacement. OpenAI's post is the primary source for the broken-task estimate. The SWE-Bench Pro paper remains useful context because it explains the benchmark's original goal: a harder, contamination-resistant testbed for enterprise-like coding work.
Community discussion is already active, especially around whether some "broken" tasks reflect unrealistic benchmark design or the messy ambiguity of real software work. That debate does not erase the audit, but it does affect how strongly teams should generalize the result.
Availability and access
There is no new model or API to enable. The action is evaluation hygiene: pause any procurement or model-ranking decision that treats SWE-Bench Pro as a clean single-number answer.
If your team uses coding agents, rerun your shortlist against your own repositories, failing tests, migration tasks, documentation edits, and review burden. Use public benchmarks to decide what to test first, then compare completed tasks, total cost, patch quality, and human review time. For a broader selection path, pair this with LinkLoot's AI agent tools guide.
Practical LinkLoot angle
The useful move is to stop asking which model has the highest public score and start asking which model-plus-harness survives your own task mix. A cheaper model that iterates well on your test suite can beat a stronger model that produces expensive patches your team will not merge.
Track four numbers during trials: completed tasks, rejected patches, reviewer minutes, and total model spend. That gives you a better operational signal than a leaderboard delta that may depend on hidden-test assumptions.
What to verify before you act
- Check whether a model vendor is reporting SWE-Bench Pro, SWE-bench Verified, Terminal-Bench, a private benchmark, or a custom scaffold.
- Ask whether the result is model-only or includes a tuned agent harness, browser, terminal, retries, and changed timeouts.
- Verify cost per completed task, not only pass rate.
- Run a small internal suite with real repository tasks and human review labels.
- Watch for a response from SWE-Bench Pro maintainers or a revised benchmark subset before treating the audit as final.
Source check
Confirmed by: OpenAI's July 8 research post says its audit found widespread SWE-Bench Pro task issues, estimates roughly 30% broken tasks, and retracts its earlier recommendation to adopt the benchmark.
Context: The SWE-Bench Pro paper describes the benchmark's intended design and task set. Hacker News discussion confirms developer attention and surfaces the live debate over whether ambiguous tasks should be fixed, filtered, or measured differently.
No. OpenAI says enough audited tasks are broken that developers should examine results carefully and that it is retracting its earlier recommendation.