The Best AI Coding Agent Fully Finishes Just 26% of Real Tasks, a New Benchmark Finds
Photo: Hippopx (public domain). Illustrative image.
What happened. A new benchmark built from real user requests shows how far AI coding agents still are from finishing the job. Co-Design Bench, posted to arXiv on 26 October 2026 by researchers from Alibaba Group and several Asian universities, gives coding agents 273 real requests to change an existing web app — the kind of revision a user asks for in a live coding session — and grades the result against 2,331 hidden acceptance criteria written by experts. The best of ten models tested, OpenAI’s GPT-6 Astra, satisfied 69.9% of the criteria on average, but fully completed only 26.4% of the tasks.
How the test works. The authors started from 37,463 deduplicated queries drawn from 26,314 real user coding sessions, and kept the 273 tasks where users had previously had to rephrase a request or roll a change back — the hard cases, by design. Each task is split into two tracks: Multimodal Editing (100 tasks), for localised changes that may involve a screenshot, image, audio, video or document; and Repository Editing (173 tasks), for coordinated changes across different parts of a project. An AI judge tests the finished app in a browser, and a second AI judge tries to refute every pass, so mock-ups and hard-coded fakes get no credit.
The results. GPT-6 Astra led all ten models, fully solving 41.0% of the multimodal tasks and 17.9% of the repository tasks. Claude Opus 5 fully completed 18.3% of tasks overall, and xAI’s Grok 4.6 came last at 6.6%. The gap between the two scores for the best model — 69.9% of requirements met, 26.4% of tasks finished — is the paper’s central finding: an agent can handle most of a task and still leave it undone. The failure mode is worst when a request mixes several kinds of change at once; on tasks requiring three or more concurrent revision types, such as layout, interaction logic and content together, performance collapsed below 18% for every model.
The harness matters as much as the model. The main comparison ran every model through Claude Code as the agent harness. Swapping the harness moved results by up to 18.9 points — Qwen3.8-27B’s repository score rose from 0.560 to 0.749 depending on the environment it ran in. The authors’ point is that “which model” is only half the question in agentic coding; the tooling wrapped around it shapes the outcome almost as much.
Why this benchmark exists. Existing tests have saturated: agentic models now score above 99% on some older benchmarks, so the differences between models can barely be measured. Co-Design Bench leaves a gap of more than 35 percentage points against WebCompass-edit, a comparable test. The Decoder notes the same pattern elsewhere: a METR study found that real acceptance of AI-generated code by project maintainers runs about 24 percentage points below SWE-bench Verified scores.
The caveats. This is an unreviewed preprint, not peer-reviewed work. All three corresponding authors, and a majority of the author list, work at Alibaba Group — a clear conflict of interest, although the top-ranked model was not an Alibaba model but OpenAI’s GPT-6 Astra, and Alibaba’s own Qwen3.8 models ranked third and fourth. The tasks were deliberately selected to be difficult, so 26.4% is a stress-test number, not the success rate on an average to-do item. And the public release of the dataset contains only 96 multimodal tasks and no repository tasks, with most requests in Chinese, so independent replication of the full result is limited for now.
What it means. For teams buying into coding agents, the honest reading is neither hype nor dismissal: the agents are genuinely useful at getting most of a change done, and genuinely unreliable at finishing it. The work that remains — checking the details, catching the one broken requirement in eight — still lands on a human. That is the gap the next generation of agents has to close.
Sources
- Wei et al. — “Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models”, arXiv:2610.35138 (26 October 2026): https://arxiv.org/abs/2610.35138?context=cs
- Hugging Face — Paper page for Co-Design Bench: https://huggingface.co/papers/2610.35138
- AI News Online — “Co-Design Bench: coding agents finish 26.4% of tasks” (8 November 2026): https://ai-news-online.com/explained/co-design-bench-coding-agents-26-percent-explained/
- MA-NO — “Co-Design Bench: Only 26.4% of Tasks Are Fully Completed” (8 November 2026): https://www.ma-no.org/en/software/development/co-design-bench-only-26-4-of-tasks-are-fully-completed
