RobustSGPO: How to Govern Search Space When Agent Harnesses Evolve

The work under review is RobustSGPO: Search-Space Control for Agent Harness Evolution, published on arXiv as 2609.09646. The original paper is linked under the title.

It proposes that SGPO, which improves agent harnesses from execution feedback yet leaves edit scope and operation unspecified, can be repaired by writing the wanted edit down explicitly, building and checking the patch, and continuing search from the incumbent or from retained snapshots. This post traces why that diagnosis appeared, what each piece changes, and what the study found across 120 tasks, 95 runs, and 7,350 candidate attempts in the AgentX brainstorming workflow.

It caught my eye because it fixes the harness, not the agent

Hello, this is Patrick. What interests me more in this paper is the choice of target. Most agent work tries a stronger model or a better line of prompt when results disappoint. This study looks one layer outward and treats the harness itself as the thing to evolve. By harness I mean the parts around the model that hand out tasks, connect tools, and set permissions. People work the same way. Hiring a strong person and building a room where that person can do strong work are two different jobs. So I leaned in from the first page. This was not a story about raising a model score. It was a story about repairing the floor the model walks on.

That phrase sounds clean, but the work itself gets messy fast. Once you decide to fix the floor, three questions arrive together. Where should you touch, how wide should the touch be, and how will you know the touch helped. I have watched harness edits happen this way in practice. Someone adjusts a prompt late at night, someone else reorders tool calls the next morning, and a week later someone touches permission settings, yet nobody can explain afterward which change mattered. So what does it take to automate that kind of fixing. That question is where RobustSGPO begins.

What happens when feedback is clear but nothing looks editable

The background is an earlier method called SGPO. It improves agent harnesses with execution feedback, using language feedback about failures as a kind of semantic gradient that points to the next edit. Readers who followed TextGrad or GEPA will recognize the family resemblance. Earlier attempts feed the next candidate, and the system learns from scores, evolution, or language critiques. At first I read SGPO as the harness version of automatic prompt repair, and that reading still seems fair.

The trouble starts after that. Even with useful feedback, search drifts when the edit scope and the operation stay open. Suppose the failure points to permission settings. You could change one sentence, or rewrite the whole permission schedule, or alter how work branches across agents. The older local update rule does not settle that choice, so the same feedback can lead to a tiny touch in one run and a strange touch elsewhere in the next. Anyone who has fixed code knows this feeling. The error message is clear, yet three files could each be the right place to edit, so your hand hovers. Search does not get wider in that moment. It gets blurry.

So here is a question I kept returning to. Should we first make feedback prose sharper, or should we first tidy the search space that receives the feedback. The authors pick the second path. Instead of polishing the critique sentence, they decide how the range and action of an edit will be controlled. I find that choice very much like engineering. It fixes the test rig before chasing the score.

Why writing down the change you want comes first

The first device in RobustSGPO is to specify the requested edit. The instruction is no longer a vague request to improve things. It states what to change, in which range, and with which action, written out before search moves. If the permission schedule needs work, the request says from which stage to which stage it should grow. If a prompt block needs work, the request says which block to add or remove. Once that promise is written, search follows the promise rather than luck. Later you can trace why an edit was tried, and you can reuse the same promise on other tasks.

This difference matters more than it looks. When our team reads a new AI architecture, we do not look at benchmark numbers alone. We ask what actually changed and what cost structure that change creates in a system, and a written edit request makes that tracing far easier. Without the request, candidates wobble in a new direction each time, and neither wins nor losses can be explained. Think of it like a written work order. Spoken instructions and written instructions produce different results on a shop floor, and the same holds here.

Would such a promise narrow search too much. I paused on that point, because fixing the range early could hide a surprising good edit. The answer in the paper leans the other way. A specification does not close search so much as stop it from scattering. You can still make many candidates inside the stated range, and a clear range makes it easier to carry the cause of failure into the next promise. The patch check I will describe next picks up right there.

The habit of checking whether the patch actually runs

The second device is to construct and check the patch. Once a request sentence exists, it is turned into an actual patch against harness code or settings, and that patch is checked for whether it can run. Runnable here means more than clean syntax. It means the harness still starts, still calls tools in order, and still carries a task to the end. Broken patches are dropped early. Only living patches move on to full evaluation.

That sounds plain on the surface. In practice it is not. In harness evolution, many nice sounding proposals fall apart at run time. Permissions widen and tools tangle, or a prompt is polished and the call order slips. If such candidates go through full evaluation, tokens and hours drain away. When the patch stage filters them, evaluation money is saved and search stays with candidates that breathe. Anyone who has run batch experiments overnight knows the feeling. When half the morning log is broken runs, the day starts with pipeline repair before any score reading.

What stayed with me is the report that this check connects directly to quality scores. Quality rose when executable edits were combined with alternative starting points, which means sentences that run beat sentences that only sound good. Of course a check cannot promise everything. A candidate can run and still feel thin. Yet at least the budget stops going to candidates that cannot run at all. So what can the saved budget buy. It can buy more starting points for search.

How search changes when you do not start only from where you stand

The third device is to continue search from the incumbent or from retained snapshots. Most evolutionary search makes the next candidate from the current best, the incumbent. That works until the incumbent walks into a dead end, after which every later try paces inside the same alley. RobustSGPO keeps past strong snapshots aside and returns to one of them when that happens. It resembles hiking. When you take a wrong turn, you do not push forward through brush. You walk back to the last fork and climb again.

The reason I like this device is that failures turn into forks rather than trash. When the incumbent overfits one task family, a stored snapshot still holds a different direction. Search then keeps choosing where to continue from, the incumbent or a retained snapshot. I read that as a map of forks rather than a notebook of wrong answers. It does not collect mistakes. It collects places from which you can restart.

This is where I would be careful not to over-read the experiment. Keeping more snapshots does not always help. Storage and restore and judgment all cost something, and returning too often keeps search from moving forward. The study reports that retention cost was measured, so the authors do not hide the price. Still the message stays clear. Returning well matters as much as choosing well. The permission schedule experiment shows how that search behaves on one concrete harness decision.

Starting with full permission was not the answer

We now enter the AgentX brainstorming flow used for evaluation. The scale covers 120 tasks, 95 runs, and 7,350 candidate attempts. The numbers are not small, but I think the shape behind them matters more than the size. The same machinery was tested on three different layers, permission scheduling and cumulative controls and task family transfer, rather than measured once for a headline.

The permission schedule result stood out first. Permission here is close to a setting that decides how far an agent may reach in one move. Giving maximum permission from the start looks kind, yet it lost. A periodic schedule that widens from 1 to 2 and then to 3 beat fixed maximum permission by 0.28 test score points. Readers of earlier multi agent work may recall how writing to a plain file versus writing to SOUL.md changed infection odds. Here too, when to open things mattered more than where to write things.

Why would that happen. With wide permission early, search feels free sooner, but odd edits enter sooner too. With staged permission, early steps firm up a narrow base and later steps try wider moves. It is close to handing a new helper one shelf to organize first, then two shelves, rather than the whole library on day one. AgentX runs brainstorming through idea drafts and shaping and selection, and that flow reacts sharply to permission settings. I did not treat 0.28 as a small gap. A schedule change alone moved the score under the same budget, so it reads as both an operations issue and a design issue.

Teams diverge even on the same token budget

If permission scheduling covers one part of the harness, the next result feels closer to the whole. Under the same budget of 20 million tokens, completion on 30 held out tasks rose from 60.0 percent to 80.0 percent, while test quality rose from 3.77 to 4.14. Completion asks whether a task was carried to the end. Quality asks whether the result was worth using. Either one alone can mislead, yet here they moved together. So I read this less as a score bump and more as a change in how work gets done.

Those numbers need their conditions read alongside them. The 20 million token budget is the condition, and it means the gain did not come from endless tries with an open wallet. From a systems perspective, the next question is obvious. Where did the tokens go that allowed more finishes and better answers for the same money. The answer offered by the study points back to the two devices above, executable edits and alternative starting points. Less money went to broken candidates, and dead ends were escaped by returning to an earlier fork.

So how far should we stretch that story. I slowed down here, because 30 tasks form a held out set and the link between finishing and quality can shift with task shape. In some families finishing is easy while quality stays flat, and in others the reverse holds. Still a 20 point move in completion is hard to dismiss. In a brainstorming flow that often stalls midway, finishing at all is a precondition for quality. The next scene, task family transfer, asks whether that gain survives a move.

Should you remember the old neighborhood after you move

A harness that lives long enough meets a shift in task family. Topics change, grading changes, or shared tools get replaced. Then a plain question returns. How much of the old way should you keep. If you drop everything and fit the new street, new work may shine while old work breaks. If you hold the old way too tightly, new work adapts slowly. The study looks at that tension through transfer runs.

The result resists a one line summary. Keeping snapshots by category softened the drop on source tasks after the shift, while keeping snapshots at random reached a higher endpoint at the destination. In other words, the way to forget less and the way to climb higher in the new place split apart. I liked that split because it felt honest. Had one retention rule won on every side, I would have doubted it. Moving house works the same way in life. Friends from the old street keep old ties alive, while a full focus on the new street can speed new settling.

So which should you pick. The answer depends on where the harness will live. When old and new task families must run side by side, category retention feels calmer. When the team moves fully to a new family, the higher endpoint of random retention catches the eye. What matters most is that retention is not free, and that cost story comes next.

It costs something, which is why I trust it more

Gathered together, the story so far could leave only a bright picture. Edits turn explicit, patches get checked, search restarts from forks, and scores rise. Yet the study does not stop there and notes the overhead of retention alongside the gains. Reports like that raise my trust rather than lower it. A report that lists costs with wins survives longer on a real floor than a report that lists wins alone.

The cost is easy to picture. Keeping snapshots and returning to them needs storage and management and judgment. Someone must decide when to store, what to drop, and from which snapshot to restart, and each decision spends compute and tokens. The fact that quality still rose means the gain survived after paying that bill. Waste fell through executable edits, and stuck paths opened through other starting points. When our team reads a new AI architecture, we apply the same yardstick. We do not ask only whether the score rose. We ask what storage and messages and restarts that score required.

From a systems perspective, the next question is obvious. How does that cost bend as scale grows. When 120 tasks become many more, or when several teams share one harness, retention rules will need to change. Some snapshots will age fast, while others will stay useful for a long time. What 95 runs and 7,350 attempts show looks less like an ending and more like a starting line. Search that measures its costs is search that can be put into operation.

Where I would start the next time I touch a harness

It is time to return to the question set aside earlier. Which comes first, prettier feedback prose or a better governed search space. After following this paper, my answer leans to the second. First write what to change, in which range, and with which action. Then check promptly whether that promise runs. When stuck, restart not from the incumbent alone but from a stored fork. The order sounds plain, yet it removes one reason after another why overnight runs break.

What I would watch next falls into three parts. First, the language of edit requests. The current form records range and action, but a wider form could record permission and tools and call order together. Second, the rule for retention. Since category retention and random retention split, we need a device that picks a retention rule from the shape of the task family. Third, movement at larger budgets and longer transfers. I want to know whether the move from 60.0 percent to 80.0 percent and from 3.77 to 4.14 keeps its shape at larger scale, and how retention cost grows there.

Back in human teams, the story turns even plainer. Doing good work is partly about talent, but it is also about how work is handed out. When to widen permission, where to write down failed tries, and to which fork a stuck team should return may sound like small chores. Piled up, they decide whether a group improves. I see agent harnesses the same way. Apart from how fast models grow smarter, there is a separate craft in governing the floor they work on. RobustSGPO shows one version of that craft with concrete hand movements. So I read this paper less like a scorecard and more like a work order. The next time I touch a harness, I will ask first whether the change we want is written down, whether that sentence is checked by running, and where our fork for return sits.