# The Resolution Hypothesis: why self-improving AI agents stall

> A study of 1,030 agent evolution runs finds the better prompts and skills are there. The tests used to pick them are too noisy to tell.

**In short:** Self-improving AI agents mostly stall because their tests are too noisy to spot a real improvement, not because they can't write better prompts or skills. In the study, picking the better of two candidates worked about as often as a coin flip.

- **1,030** evolution runs tested, across 7 methods and 6 models
- **49.7%** of the time, picking by score chose the better version. A coin flip: 50%.
- **62.1%** of the versions kept were actually worse than before
- **17.6 → 0.8** points: the median big gain, once the test was fixed

Explains "The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements" by Yuanhao Zeng, Shaocheng Lan, Ao Lu, Kuang Yan, Heng Lin, Xinang Xu, Yuan Luo, Yueting Wang, Yexin Li, Ruo Shan, Kan Ren (ShanghaiTech University, ByteDance, BIGAI, SUSTech). arXiv 2611.23068, 2026-11-17. Original paper: https://arxiv.org/abs/2611.23068 (CC BY 4.0).

An explainer by Venkat R. Murty: https://designervenkat.online/ai-papers/resolution-hypothesis

## What this paper explains

- What agent evolution is, and why so many labs are betting on it
- Why test scores can't tell a real improvement from luck
- Why rankings between methods in papers often rest on chance
- How one added sentence explains the biggest reported gains
- What to change if you build or test AI agents yourself

## The idea in one picture

*Agent evolution as a loop. The paper finds step 3 is where it breaks: a lucky score looks the same as a real gain.*

1. **Propose:** An AI writes new versions of the agent's prompt, skills or memory.
2. **Test:** Each version runs on a set of tasks and gets a score. Scores wobble from run to run.
3. **Pick:** The version with the highest score is kept for the next round.
4. **Repeat:** Round after round, lucky picks pile up and runs drift apart.

## What is agent evolution?

Most AI agents run on a model whose weights never change. You can't retrain it, but you can change what you hand it: the instructions in its prompt, a library of skills it can call, notes it keeps in memory, or the scaffolding around it (the harness that runs the model, gives it tools and decides what it sees).

Agent evolution automates that tuning. The paper describes it as a loop: one AI proposes new versions of these pieces, the agent is tested on a set of tasks with each version, and the version that scores higher is kept. Run the loop for several rounds and, in theory, the agent keeps getting better at its job without anyone touching the model.

It is an attractive idea for anyone who runs agents in production. A system that improves its own workflows while you sleep would save a lot of manual prompt tweaking. The trouble is that the results have been hard to trust.

- **Artifact:** Anything the loop changes outside the model: a prompt, a skill, a memory note or the harness.
- **Proposer:** The part that writes new candidate artifacts.
- **Selector:** The part that decides which candidate to keep, usually by test score.
- **Benchmark:** A fixed set of tasks used to score an agent, such as Terminal-Bench for command-line work.

## What problem does this paper look at?

The authors start from a puzzle. The same evolution methods look strong in one paper and weak in the next. On some benchmarks, Terminal-Bench among them, evolution keeps stalling. On others, every new method claims to beat the one before. Nobody could predict when evolution would help.

Their answer is that the problem sits in the measuring, not in the ideas. If the score you use to judge a candidate moves around by more than the improvement you are looking for, you can't tell a better candidate from a lucky one. Pick by that score and you are partly picking at random.

## What is the Resolution Hypothesis?

The paper puts it in one line: the hard part of agent evolution is identifying which changes help, not proposing them.

It borrows the word resolution from cameras and microscopes. The resolution of a test is the smallest real improvement it can reliably tell apart from noise. A blurry photo can't show a hairline crack. A noisy benchmark can't show a 2-point gain.

Why are agent tests so noisy? Because the model's weights are frozen, a new prompt or skill can only help by nudging how the model behaves when it runs. And the same agent, on the same task, takes a slightly different path every time. Two runs of the exact same setup can land several points apart.

> **Tip:** Think of a scale that drifts by 3 kg. It is fine for weighing a suitcase. It is useless for checking whether you lost half a kilo this week.

*Illustration: A sketch of the idea (made-up values, not data from the paper). Same four candidates on both sides. With noisy scores, the square scores highest by luck and gets kept, though it is really the worst. With clear scores, the circle wins, as it should.*

- **Noisy scores:** Keeps a worse version
- **Clear scores:** Keeps the best version

## How did they test it?

They went big. Where most papers report one evolution run, this study ran each method 8 times per model and benchmark and scored each result up to 16 times.

- **7 methods:** GEPA for prompts; SkillOpt-Lite and Trace2Skill for skills; ReasoningBank for memory; HarnessLens, Meta-Harness and AHE for the harness.
- **6 models:** GPT-5.4 nano, GPT-5.6 Luna and Terra, DeepSeek-V4-Flash, Kimi-K3 and GLM-5.2.
- **4 + 9 benchmarks:** ALFWorld, SpreadsheetBench, OfficeQA and Terminal-Bench v2.1, plus nine more where they re-checked earlier claims.

- **7** ways to evolve an agent, tested side by side
- **6** AI models, from GPT-5.4 nano to GLM-5.2
- **13** benchmarks: 4 main ones and 9 re-checked
- **2.1M** test runs of the agent, all scored

In all: 1,030 independent evolution runs, over 2.1 million test runs of the agent, over 10 million model calls and 400,000 CPU core hours. The authors call it the largest reproducibility study of agent evolution so far, as far as they know.

They split the noise into two kinds. Evaluation noise is how much the score of one fixed version moves when you test it again. Run-to-run variance is how much two separate evolution runs of the same method end up differing. Both matter, for different questions.

## Can we trust the rankings in papers?

Mostly not, on today's practice. The noise in a single score was 1.0 to 3.7 percentage points on SpreadsheetBench, OfficeQA and ALFWorld, and 4.5 points on Terminal-Bench.

Now compare that with the margins papers report. The authors looked at 171 results in the main tables of nine agent-evolution papers. The new method beat its strongest rival by a median of 3.4 points, with the middle half of results between 1.2 and 6.6 points. None of those results reported a variance estimate, and only 6 averaged more than one evolution run.

**Chart: Of 171 published results, how many averaged more than one run?**

6 of 171: averaged more than one evolution run. The rest: came from a single run.

One dot per result in nine papers. None of them reported how much the scores wobble. Source: [Section 4.1](https://arxiv.org/html/2611.23068v1#S4.SS1)

So the typical win is about the size of the wobble. To detect a 3.4-point gap reliably, you would need about 5 runs per method on SpreadsheetBench, 10 on OfficeQA, 19 on ALFWorld and 23 on Terminal-Bench.

With one run per method, a results table picked the truly best of the four text-based methods only 33% of the time on Terminal-Bench. A random guess would get 25%.

**Chart: How many runs does each method need to show a typical 3.4-point win?**

- SpreadsheetBench: 5 runs
- OfficeQA: 10 runs
- ALFWorld: 19 runs
- Terminal-Bench: 23 runs
- Most papers: 1 run

Most papers run each method once. Spotting a typical win needs 5 to 23 runs. Source: [Section 4.1](https://arxiv.org/html/2611.23068v1#S4.SS1)

**Chart: How often does a one-run results table name the truly best method?**

- Other 3 benchmarks: 51 to 73%
- Terminal-Bench: 33%
- Random guess: 25%

On the noisiest benchmark, a single run barely beats guessing. Source: [Section 4.1](https://arxiv.org/html/2611.23068v1#S4.SS1)

### Where do the big reported gains come from?

Some benchmarks do show gains of 10 to 40 points, far above the noise. That looks like proof that evolution works. The authors dug into the evolved prompts and skills line by line to see what was doing the work.

Usually it was one sentence. Adding just the single most responsible rule to the plain, unevolved agent recovered a median of 90% of the gain, across 27 model and benchmark pairs with a reported gain above 3 points.

That rule was typically a shortcut for the test itself: matching the one answer format a grader accepts, or fitting a version of the benchmark that differs from the official one. When the authors repaired those tests, grading answers by meaning instead of exact wording, the median gain fell from 17.6 points to 0.8.

**Chart: What happens to the big gains once the test is fixed?**

as reported: 17.6 pts → test fixed: 0.8 pts

Median gain on benchmarks that had a shortcut. Fix the grader and the gain almost disappears. Source: [Section 4.2](https://arxiv.org/html/2611.23068v1#S4.SS2)

This fits the hypothesis from the other side. Evolution does keep an improvement when it is big enough to see through the noise. A trick that lifts almost every task at once is exactly that kind of improvement, even when it teaches the agent nothing useful.

## Are better versions there at all?

Yes. To check the proposer on its own, the authors built a fixed pool of candidate skills from 10 Terminal-Bench training tasks and tested every one on 10 different tasks, 16 times each: 16,000 runs in total.

Keeping the best skill for each task really did raise success as the pool grew, and it stayed clearly above a control curve built to show what luck alone would produce. Single skills that helped across all 10 tasks existed too: across the pool, skills differed by more than 15 percentage points.

- **16,000** test runs of candidate skills on 10 new tasks
- **15+ pts** between the best and the worst skill in the pool

So the claim that AI can't write useful improvements does not hold up here. Good candidates are being produced. They are just not being kept.

## Why can't the agent pick the good ones?

Next they tested the selector alone, on a second fixed pool of skills spread across 11 Terminal-Bench tasks of different difficulty. Each selector saw one test run per skill, which is what real evolution loops do, and its choice was checked against the true quality measured on many more runs.

1. **Coin-flip accuracy:** Choosing by test score picked the better of two skills 49.7% of the time. Random choice gets 50%.
2. **Bad versions get through:** Skills that made things worse were 30.5% of the pool, but 62.1% of the ones the selector accepted.
3. **Score says nothing:** The link between one test score and true quality was 0.054 on a scale where 0 means no link at all.
4. **AI judges don't fix it:** Having a model read the full run and judge it scored 45.4% to 49.0%. The judge adds its own noise on top.

**Chart: How often does each way of choosing pick the better of two versions?**

- Random choice: 50.0%
- Test score, 1 run: 49.7%
- Test score, 2 runs: 48.6%
- AI judge, side by side: 49.0%
- AI judge, ACE: 46.9%
- AI judge, one at a time: 45.4%
- Coin flip: 50%

Every method lands at or below a coin flip. Source: [Table 1](https://arxiv.org/html/2611.23068v1#S5.T1)

**Chart: How many versions were worse than the one before?**

- of all versions in the pool: 30.5%
- of the versions picking by score kept: 62.1%

Picking by score kept bad versions about twice as often as they appear. Source: [Section 5.2](https://arxiv.org/html/2611.23068v1#S5.SS2)

Giving the selectors more candidates did not help either. A perfect selector kept finding better skills as the pool grew to 100. Every real selector stayed flat, close to random.

### What happens over many rounds?

Each round keeps whatever happened to score highest, so the loop keeps accepting lucky draws. The authors re-tested every version from eight GEPA runs and eight SkillOpt-Lite runs on Terminal-Bench, eight times per task instead of once.

Inside the loop, the recorded gain climbed to 6.9 to 9.0 points. On careful re-testing, the same versions ended between 0.4 points better and 3.2 points worse than where they started. The paper sums it up: "In-loop scores improve under this process; agent capability does not."

**Chart: Gain shown inside the loop, and the same versions re-tested**

- Inside the loop: 6.9 to 9.0 pts
- Re-tested 8 times: -3.2 to +0.4 pts

The loop's own scores climb. Careful re-testing shows no real gain. Source: [Section 5.3](https://arxiv.org/html/2611.23068v1#S5.SS3)

The random picks also push separate runs apart. All runs started from the same version, yet by the last round they differed by about as much as the margins that separate methods in published papers. That is the ranking instability from earlier, now explained.

## What do the authors suggest?

- **Check behaviour, not only pass/fail:** Add cheap checks on what the agent does along the way, such as whether it runs a test after editing a file. In the authors' early tests, one such check (whether the agent left error output visible when it verified its work) was 13 to 57 times less noisy than a final pass or fail.
- **Test the parts separately:** Measure how good the proposer's best ideas are, and separately how well the selector picks from a fixed pool. Each is easier to measure than the whole loop.
- **Report the noise:** Publish how much scores move on re-testing and between runs, so readers can see whether a claimed win is bigger than the wobble.

The wider point: building better feedback may matter as much as building cleverer evolution methods.

## What does this mean if you build AI agents?

My take, as someone who builds agent workflows for clients. None of this is in the paper itself; it is how I would apply it.

1. **Never trust one run:** Before you keep a new prompt or skill, run the old and new versions several times on the same tasks and compare the averages.
2. **Know your wobble:** Run the same version a few times first. If scores move by 4 points, a 2-point win means nothing yet.
3. **Read the winning change:** When a change gives a big jump, read it. If it only teaches the agent your grader's format, it will not help real users.
4. **Add process checks:** Log simple behaviours, like 'ran the tests' or 'checked its answer'. They catch real improvement sooner than a final score.

## What are the limits of this study?

The authors name two. The study only covers agents whose model weights stay frozen; methods that retrain the model may behave differently. And the candidate-level evidence comes from Terminal-Bench v2.1 only, so other task types may differ.

One more to keep in mind: this is a preprint, not yet peer reviewed, and the authors say the code and run data will be released once the paper is accepted.

## Sources checked

- Agent evolution changes things outside a frozen model: "Agent evolution improves frozen LLM agents by refining persistent external artifacts such as prompts, skills, memories, and entire agent harnesses." ([source](https://arxiv.org/html/2611.23068v1#S1))
- The hypothesis: "The bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them." ([source](https://arxiv.org/html/2611.23068v1#S1))
- What resolution means: "We define the resolution of an evaluation pipeline as the smallest true improvement it can reliably distinguish from noise." ([source](https://arxiv.org/html/2611.23068v1#S1))
- Size of the study: "totaling 1,030 independent evolution runs and over 2.1 million test trajectories" ([source](https://arxiv.org/html/2611.23068v1#S1))
- Compute used: "In total, the experiments consumed over 10M LLM calls and 400K CPU core hours." ([source](https://arxiv.org/html/2611.23068v1#S3))
- Noise in a single score: "ranges from 1.0 to 3.7 percentage points (pp) on SpreadsheetBench, OfficeQA, and ALFWorld, and reaches 4.5 pp on Terminal-Bench" ([source](https://arxiv.org/html/2611.23068v1#S4))
- 171 results from nine papers: "Across 171 benchmark–model results in the main tables of nine agent-evolution papers" ([source](https://arxiv.org/html/2611.23068v1#S4.SS1))
- 7 methods, 6 models, 4 + 9 benchmarks: "the study covers seven agent evolution methods over six models and four agent benchmarks, plus nine further benchmarks" ([source](https://arxiv.org/html/2611.23068v1#S1))
- 16,000 test runs of candidate skills: "16 independent rollouts per skill–task pair (16,000 rollouts total" ([source](https://arxiv.org/html/2611.23068v1#S5.SS1))
- Typical published margin: "the proposed method leads its strongest competitor by a median of 3.4 pp, with an interquartile range of 1.2 to 6.6 pp." ([source](https://arxiv.org/html/2611.23068v1#S4))
- Few papers report variance: "None of these results reports a variance estimate, and only 6 average over more than one evolution run." ([source](https://arxiv.org/html/2611.23068v1#S4))
- Runs needed to see a 3.4-point gap: "At the median reported margin of 3.4 pp, this means about 5 runs on SpreadsheetBench, 10 on OfficeQA, 19 on ALFWorld, and 23 on Terminal-Bench" ([source](https://arxiv.org/html/2611.23068v1#S4))
- Other benchmarks: best method named 51% to 73% of the time: "picks the best of the four text-artifact methods only 51% to 73% of the time on SpreadsheetBench, OfficeQA, and ALFWorld" ([source](https://arxiv.org/html/2611.23068v1#S4.SS1))
- Two-run selection: 48.6%: "The native parent-selection operators of SkillOpt-Lite and GEPA, which observe two rollouts per artifact rather than one, achieve 48.6% pairwise accuracy." ([source](https://arxiv.org/html/2611.23068v1#S5.SS2))
- AI judge one at a time: 45.4%: "LLM judge (pointwise) 45.4" ([source](https://arxiv.org/html/2611.23068v1#S5.T1))
- AI judge side by side: 49.0%: "LLM judge (pairwise) 49.0" ([source](https://arxiv.org/html/2611.23068v1#S5.T1))
- AI judge (ACE): 46.9%: "LLM judge (ACE) 46.9" ([source](https://arxiv.org/html/2611.23068v1#S5.T1))
- Single-run tables barely beat guessing: "and only 33% on Terminal-Bench, where evaluation noise is largest; a random guess gets 25%." ([source](https://arxiv.org/html/2611.23068v1#S4))
- One sentence recovers most of the big gains: "across the 27 model–benchmark pairs with a reported gain above 3 pp, it recovers a median of 90% of the gain." ([source](https://arxiv.org/html/2611.23068v1#S4))
- Repairing the test removes the gain: "The median gain falls from 17.6 to 0.8 pp" ([source](https://arxiv.org/html/2611.23068v1#S4))
- Good skills exist: "individual skills in the Proposal Pool span more than 15 percentage points" ([source](https://arxiv.org/html/2611.23068v1#S5))
- Picking by score is a coin flip: "execution-based selection prefers the better artifact only 49.7% of the time, statistically indistinguishable from the 50% achieved by random selection." ([source](https://arxiv.org/html/2611.23068v1#S5))
- Bad versions get accepted: "Regressions make up 30.5% of the pool but 62.1% of accepted candidates" ([source](https://arxiv.org/html/2611.23068v1#S5))
- Score and quality barely related: "The rank correlation between a single execution score and posterior artifact quality is 0.054, essentially zero." ([source](https://arxiv.org/html/2611.23068v1#S5))
- AI judges don't help: "No judge exhibits reliable pairwise accuracy." ([source](https://arxiv.org/html/2611.23068v1#S5))
- In-loop gains are mostly luck: "In-loop gains grow to 6.9 to 9.0 pp over the course of evolution" ([source](https://arxiv.org/html/2611.23068v1#S5.SS3))
- Re-tested, the gains vanish: "re-evaluated gains remain between +0.4 and -3.2 pp" ([source](https://arxiv.org/html/2611.23068v1#S5.SS3))
- Scores rise, ability doesn't: "In-loop scores improve under this process; agent capability does not." ([source](https://arxiv.org/html/2611.23068v1#S5.SS3))
- Suggested process checks: "whether the agent runs a test after editing a file" ([source](https://arxiv.org/html/2611.23068v1#S6))
- Error-visibility checks are 13 to 57 times less noisy: "unmasked-diagnostic check rates exhibit 13\times – 57\times lower evaluation noise than terminal pass/fail verdicts" ([source](https://arxiv.org/html/2611.23068v1#S6))
- Limit: frozen weights only: "We study agent evolution where model weights remain frozen and evolution operates only on external state (prompts, memories, skills, harnesses)." ([source](https://arxiv.org/html/2611.23068v1#Sx1))
- Limit: one benchmark for candidate tests: "The candidate-level evidence is evaluated on Terminal-Bench v2.1 only" ([source](https://arxiv.org/html/2611.23068v1#Sx1))
- Data release: "All reproducible materials and trajectory data will be released upon the acceptance of this paper." ([source](https://arxiv.org/html/2611.23068v1#S1))
