Yuanhao Zeng, Shaocheng Lan and 9 others · ShanghaiTech University, ByteDance, BIGAI, SUSTech · arXiv 2611.23068 · 17 November 2026
The Resolution Hypothesis: why self-improving AI agents stall
A study of 1,030 agent evolution runs finds the better prompts and skills are there. The tests used to pick them are too noisy to tell.
In short
Self-improving AI agents mostly stall because their tests are too noisy to spot a real improvement, not because they can't write better prompts or skills. In the study, picking the better of two candidates worked about as often as a coin flip.
13 min readPaper explainedCC BY 4.0
1,030
evolution runs tested, across 7 methods and 6 models
49.7%
of the time, picking by score chose the better version. A coin flip: 50%.
62.1%
of the versions kept were actually worse than before
17.6 → 0.8
points: the median big gain, once the test was fixed
propose
An AI writes new versions of the agent's prompt, skills or memory.
test
Each version runs on a set of tasks and gets a score. Scores wobble from run to run.
pick
The version with the highest score is kept for the next round.
breaks here
repeat
Round after round, lucky picks pile up and runs drift apart.
What this paper explains
- What agent evolution is, and why so many labs are betting on it
- Why test scores can't tell a real improvement from luck
- Why rankings between methods in papers often rest on chance
- How one added sentence explains the biggest reported gains
- What to change if you build or test AI agents yourself
What is agent evolution?
Most AI agents run on a model whose weights never change. You can't retrain it, but you can change what you hand it: the instructions in its prompt, a library of skills it can call, notes it keeps in memory, or the scaffolding around it (the harness that runs the model, gives it tools and decides what it sees).
Agent evolution automates that tuning. The paper describes it as a loop: one AI proposes new versions of these pieces, the agent is tested on a set of tasks with each version, and the version that scores higher is kept. Run the loop for several rounds and, in theory, the agent keeps getting better at its job without anyone touching the model.
It is an attractive idea for anyone who runs agents in production. A system that improves its own workflows while you sleep would save a lot of manual prompt tweaking. The trouble is that the results have been hard to trust.
Key words
- Artifact
- Anything the loop changes outside the model: a prompt, a skill, a memory note or the harness.
- Proposer
- The part that writes new candidate artifacts.
- Selector
- The part that decides which candidate to keep, usually by test score.
- Benchmark
- A fixed set of tasks used to score an agent, such as Terminal-Bench for command-line work.
What problem does this paper look at?
The authors start from a puzzle. The same evolution methods look strong in one paper and weak in the next. On some benchmarks, Terminal-Bench among them, evolution keeps stalling. On others, every new method claims to beat the one before. Nobody could predict when evolution would help.
Their answer is that the problem sits in the measuring, not in the ideas. If the score you use to judge a candidate moves around by more than the improvement you are looking for, you can't tell a better candidate from a lucky one. Pick by that score and you are partly picking at random.
What is the Resolution Hypothesis?
The paper puts it in one line: the hard part of agent evolution is identifying which changes help, not proposing them.
It borrows the word resolution from cameras and microscopes. The resolution of a test is the smallest real improvement it can reliably tell apart from noise. A blurry photo can't show a hairline crack. A noisy benchmark can't show a 2-point gain.
Why are agent tests so noisy? Because the model's weights are frozen, a new prompt or skill can only help by nudging how the model behaves when it runs. And the same agent, on the same task, takes a slightly different path every time. Two runs of the exact same setup can land several points apart.
Tip
Think of a scale that drifts by 3 kg. It is fine for weighing a suitcase. It is useless for checking whether you lost half a kilo this week.
Noisy scores
Keeps a worse version
Clear scores
Keeps the best version
▮ bar: measured score⌶ whisker: how much it wobbles○ ring: true quality
A sketch of the idea (made-up values, not data from the paper). Same four candidates on both sides. With noisy scores, the square scores highest by luck and gets kept, though it is really the worst. With clear scores, the circle wins, as it should.
How did they test it?
They went big. Where most papers report one evolution run, this study ran each method 8 times per model and benchmark and scored each result up to 16 times.
7 methods
GEPA for prompts; SkillOpt-Lite and Trace2Skill for skills; ReasoningBank for memory; HarnessLens, Meta-Harness and AHE for the harness.
6 models
GPT-5.4 nano, GPT-5.6 Luna and Terra, DeepSeek-V4-Flash, Kimi-K3 and GLM-5.2.
4 + 9 benchmarks
ALFWorld, SpreadsheetBench, OfficeQA and Terminal-Bench v2.1, plus nine more where they re-checked earlier claims.
7
ways to evolve an agent, tested side by side
6
AI models, from GPT-5.4 nano to GLM-5.2
13
benchmarks: 4 main ones and 9 re-checked
2.1M
test runs of the agent, all scored
In all: 1,030 independent evolution runs, over 2.1 million test runs of the agent, over 10 million model calls and 400,000 CPU core hours. The authors call it the largest reproducibility study of agent evolution so far, as far as they know.
They split the noise into two kinds. Evaluation noise is how much the score of one fixed version moves when you test it again. Run-to-run variance is how much two separate evolution runs of the same method end up differing. Both matter, for different questions.
Can we trust the rankings in papers?
Mostly not, on today's practice. The noise in a single score was 1.0 to 3.7 percentage points on SpreadsheetBench, OfficeQA and ALFWorld, and 4.5 points on Terminal-Bench.
Now compare that with the margins papers report. The authors looked at 171 results in the main tables of nine agent-evolution papers. The new method beat its strongest rival by a median of 3.4 points, with the middle half of results between 1.2 and 6.6 points. None of those results reported a variance estimate, and only 6 averaged more than one evolution run.
Of 171 published results, how many averaged more than one run?
6 averaged more than one evolution run165 came from a single run
So the typical win is about the size of the wobble. To detect a 3.4-point gap reliably, you would need about 5 runs per method on SpreadsheetBench, 10 on OfficeQA, 19 on ALFWorld and 23 on Terminal-Bench.
With one run per method, a results table picked the truly best of the four text-based methods only 33% of the time on Terminal-Bench. A random guess would get 25%.
How many runs does each method need to show a typical 3.4-point win?
- SpreadsheetBench5 runs
- OfficeQA10 runs
- ALFWorld19 runs
- Terminal-Bench23 runs
Most papers: 1 run
How often does a one-run results table name the truly best method?
- Other 3 benchmarks51 to 73%
- Terminal-Bench33%
Random guess: 25%
Where do the big reported gains come from?
Some benchmarks do show gains of 10 to 40 points, far above the noise. That looks like proof that evolution works. The authors dug into the evolved prompts and skills line by line to see what was doing the work.
Usually it was one sentence. Adding just the single most responsible rule to the plain, unevolved agent recovered a median of 90% of the gain, across 27 model and benchmark pairs with a reported gain above 3 points.
That rule was typically a shortcut for the test itself: matching the one answer format a grader accepts, or fitting a version of the benchmark that differs from the official one. When the authors repaired those tests, grading answers by meaning instead of exact wording, the median gain fell from 17.6 points to 0.8.
What happens to the big gains once the test is fixed?
This fits the hypothesis from the other side. Evolution does keep an improvement when it is big enough to see through the noise. A trick that lifts almost every task at once is exactly that kind of improvement, even when it teaches the agent nothing useful.
Are better versions there at all?
Yes. To check the proposer on its own, the authors built a fixed pool of candidate skills from 10 Terminal-Bench training tasks and tested every one on 10 different tasks, 16 times each: 16,000 runs in total.
Keeping the best skill for each task really did raise success as the pool grew, and it stayed clearly above a control curve built to show what luck alone would produce. Single skills that helped across all 10 tasks existed too: across the pool, skills differed by more than 15 percentage points.
16,000
test runs of candidate skills on 10 new tasks
15+ pts
between the best and the worst skill in the pool
So the claim that AI can't write useful improvements does not hold up here. Good candidates are being produced. They are just not being kept.
Why can't the agent pick the good ones?
Next they tested the selector alone, on a second fixed pool of skills spread across 11 Terminal-Bench tasks of different difficulty. Each selector saw one test run per skill, which is what real evolution loops do, and its choice was checked against the true quality measured on many more runs.
- 1
Coin-flip accuracy
Choosing by test score picked the better of two skills 49.7% of the time. Random choice gets 50%.
- 2
Bad versions get through
Skills that made things worse were 30.5% of the pool, but 62.1% of the ones the selector accepted.
- 3
Score says nothing
The link between one test score and true quality was 0.054 on a scale where 0 means no link at all.
- 4
AI judges don't fix it
Having a model read the full run and judge it scored 45.4% to 49.0%. The judge adds its own noise on top.
How often does each way of choosing pick the better of two versions?
- Random choice50.0%
- Test score, 1 run49.7%
- Test score, 2 runs48.6%
- AI judge, side by side49.0%
- AI judge, ACE46.9%
- AI judge, one at a time45.4%
Coin flip: 50%
How many versions were worse than the one before?
- of all versions in the pool
- of the versions picking by score kept
Giving the selectors more candidates did not help either. A perfect selector kept finding better skills as the pool grew to 100. Every real selector stayed flat, close to random.
What happens over many rounds?
Each round keeps whatever happened to score highest, so the loop keeps accepting lucky draws. The authors re-tested every version from eight GEPA runs and eight SkillOpt-Lite runs on Terminal-Bench, eight times per task instead of once.
Inside the loop, the recorded gain climbed to 6.9 to 9.0 points. On careful re-testing, the same versions ended between 0.4 points better and 3.2 points worse than where they started. The paper sums it up: "In-loop scores improve under this process; agent capability does not."
Gain shown inside the loop, and the same versions re-tested
- Inside the loop6.9 to 9.0 pts
- Re-tested 8 times-3.2 to +0.4 pts
The random picks also push separate runs apart. All runs started from the same version, yet by the last round they differed by about as much as the margins that separate methods in published papers. That is the ranking instability from earlier, now explained.
What do the authors suggest?
Check behaviour, not only pass/fail
Add cheap checks on what the agent does along the way, such as whether it runs a test after editing a file. In the authors' early tests, one such check (whether the agent left error output visible when it verified its work) was 13 to 57 times less noisy than a final pass or fail.
Test the parts separately
Measure how good the proposer's best ideas are, and separately how well the selector picks from a fixed pool. Each is easier to measure than the whole loop.
Report the noise
Publish how much scores move on re-testing and between runs, so readers can see whether a claimed win is bigger than the wobble.
The wider point: building better feedback may matter as much as building cleverer evolution methods.
What does this mean if you build AI agents?
My take, as someone who builds agent workflows for clients. None of this is in the paper itself; it is how I would apply it.
- 1
Never trust one run
Before you keep a new prompt or skill, run the old and new versions several times on the same tasks and compare the averages.
- 2
Know your wobble
Run the same version a few times first. If scores move by 4 points, a 2-point win means nothing yet.
- 3
Read the winning change
When a change gives a big jump, read it. If it only teaches the agent your grader's format, it will not help real users.
- 4
Add process checks
Log simple behaviours, like 'ran the tests' or 'checked its answer'. They catch real improvement sooner than a final score.
What are the limits of this study?
The authors name two. The study only covers agents whose model weights stay frozen; methods that retrain the model may behave differently. And the candidate-level evidence comes from Terminal-Bench v2.1 only, so other task types may differ.
One more to keep in mind: this is a preprint, not yet peer reviewed, and the authors say the code and run data will be released once the paper is accepted.
Sources checked
Each fact on this page, and the exact words in the paper that back it. All 33 quotes were matched against the official text on 19 November 2026.
Agent evolution changes things outside a frozen model
“Agent evolution improves frozen LLM agents by refining persistent external artifacts such as prompts, skills, memories, and entire agent harnesses.”
See it in the paper ↗The hypothesis
“The bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them.”
See it in the paper ↗What resolution means
“We define the resolution of an evaluation pipeline as the smallest true improvement it can reliably distinguish from noise.”
See it in the paper ↗Size of the study
“totaling 1,030 independent evolution runs and over 2.1 million test trajectories”
See it in the paper ↗Compute used
“In total, the experiments consumed over 10M LLM calls and 400K CPU core hours.”
See it in the paper ↗Noise in a single score
“ranges from 1.0 to 3.7 percentage points (pp) on SpreadsheetBench, OfficeQA, and ALFWorld, and reaches 4.5 pp on Terminal-Bench”
See it in the paper ↗171 results from nine papers
“Across 171 benchmark–model results in the main tables of nine agent-evolution papers”
See it in the paper ↗7 methods, 6 models, 4 + 9 benchmarks
“the study covers seven agent evolution methods over six models and four agent benchmarks, plus nine further benchmarks”
See it in the paper ↗16,000 test runs of candidate skills
“16 independent rollouts per skill–task pair (16,000 rollouts total”
See it in the paper ↗Typical published margin
“the proposed method leads its strongest competitor by a median of 3.4 pp, with an interquartile range of 1.2 to 6.6 pp.”
See it in the paper ↗Few papers report variance
“None of these results reports a variance estimate, and only 6 average over more than one evolution run.”
See it in the paper ↗Runs needed to see a 3.4-point gap
“At the median reported margin of 3.4 pp, this means about 5 runs on SpreadsheetBench, 10 on OfficeQA, 19 on ALFWorld, and 23 on Terminal-Bench”
See it in the paper ↗Other benchmarks: best method named 51% to 73% of the time
“picks the best of the four text-artifact methods only 51% to 73% of the time on SpreadsheetBench, OfficeQA, and ALFWorld”
See it in the paper ↗Two-run selection: 48.6%
“The native parent-selection operators of SkillOpt-Lite and GEPA, which observe two rollouts per artifact rather than one, achieve 48.6% pairwise accuracy.”
See it in the paper ↗AI judge one at a time: 45.4%
“LLM judge (pointwise) 45.4”
See it in the paper ↗AI judge side by side: 49.0%
“LLM judge (pairwise) 49.0”
See it in the paper ↗AI judge (ACE): 46.9%
“LLM judge (ACE) 46.9”
See it in the paper ↗Single-run tables barely beat guessing
“and only 33% on Terminal-Bench, where evaluation noise is largest; a random guess gets 25%.”
See it in the paper ↗One sentence recovers most of the big gains
“across the 27 model–benchmark pairs with a reported gain above 3 pp, it recovers a median of 90% of the gain.”
See it in the paper ↗Repairing the test removes the gain
“The median gain falls from 17.6 to 0.8 pp”
See it in the paper ↗Good skills exist
“individual skills in the Proposal Pool span more than 15 percentage points”
See it in the paper ↗Picking by score is a coin flip
“execution-based selection prefers the better artifact only 49.7% of the time, statistically indistinguishable from the 50% achieved by random selection.”
See it in the paper ↗Bad versions get accepted
“Regressions make up 30.5% of the pool but 62.1% of accepted candidates”
See it in the paper ↗Score and quality barely related
“The rank correlation between a single execution score and posterior artifact quality is 0.054, essentially zero.”
See it in the paper ↗AI judges don't help
“No judge exhibits reliable pairwise accuracy.”
See it in the paper ↗In-loop gains are mostly luck
“In-loop gains grow to 6.9 to 9.0 pp over the course of evolution”
See it in the paper ↗Re-tested, the gains vanish
“re-evaluated gains remain between +0.4 and -3.2 pp”
See it in the paper ↗Scores rise, ability doesn't
“In-loop scores improve under this process; agent capability does not.”
See it in the paper ↗Suggested process checks
“whether the agent runs a test after editing a file”
See it in the paper ↗Error-visibility checks are 13 to 57 times less noisy
“unmasked-diagnostic check rates exhibit 13\times – 57\times lower evaluation noise than terminal pass/fail verdicts”
See it in the paper ↗Limit: frozen weights only
“We study agent evolution where model weights remain frozen and evolution operates only on external state (prompts, memories, skills, harnesses).”
See it in the paper ↗Limit: one benchmark for candidate tests
“The candidate-level evidence is evaluated on Terminal-Bench v2.1 only”
See it in the paper ↗Data release
“All reproducible materials and trajectory data will be released upon the acceptance of this paper.”
See it in the paper ↗
Based on “The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements” by Yuanhao Zeng, Shaocheng Lan, Ao Lu, Kuang Yan, Heng Lin, Xinang Xu, Yuan Luo, Yueting Wang, Yexin Li, Ruo Shan, Kan Ren, shared under CC BY 4.0. The words here are my own summary; all credit for the research goes to the authors.
Last updated on 19 November 2026
New AI papers, explained every week
I read the official papers from the big labs and research groups and explain the ones that matter for people building with AI. Want help applying one to your own product? Book a short call.