HarnessDev: Can LLMs create and evolve their own agent harness?

Patrick Rho · AI Research

The work under review is HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness, identifier arXiv 2609.01437. The claim fits in one sentence. Instead of leaning on human designed agent frameworks, it asks whether a language model can directly generate harness code and evolve the harness by keeping the variants that perform on executed tasks. The link to the original paper appears under the title of this post.

What interests me more is the direction of the question. Most agent research looks inward at the model. A larger model, longer reasoning, better tool use. HarnessDev looks outward at the program around the model, the code that calls tools, orders steps, manages memory, and recovers from failure, and asks whether the model can write that code itself. I liked that turn. In practice, agents often stall not because the model is weak but because the surrounding program is fragile.

Why I picked up this paper

Anyone who has operated an agent knows the same late night. The agent worked in the demo and then stopped in a strange place once it met real work. A tool changed and the parser broke, a call loop never ended, an error message got swallowed somewhere, and the logs showed the model output was fine while the code carrying it to the next step had collapsed. Model scores went up while felt reliability stayed flat. That gap is the reason this paper caught my eye. The bottleneck is not only inside the model.

When my team and I look at a new AI architecture, we do not look at benchmark numbers alone. We look at what actually changed and what cost structure that change creates in a system. HarnessDev puts that outer layer under study. It treats the harness not as a part a person fixes by hand each time but as material for generation and selection. It asks what happens when the model writes code, the code runs tasks, and the code with good results survives. The question is practical, and if the answer holds, the way teams build agents could change.

This is where I would be careful not to over-read the experiment. The evidence I had for this piece sits at the level of a research outline centered on the arXiv identifier. Fine method detail and exact numbers need direct consultation of the original text. So treat this post as an interpretation rather than a firm report of figures. It organizes why the idea appeared, how the structure runs, where it gets hard, and what to check next. At this stage, getting the structure right matters more than repeating numbers.

The harness is another program outside the model

The word harness gets used loosely. In this post it means the whole execution program wrapped around model calls. When a request arrives, the code that decides the order of model calls, which tools are exposed, how tool results return, where intermediate memory lives and how much stays, and how many retries happen on failure is all harness. It is less like a single prompt and more like a small operating system.

Picture a research agent that searches the web and writes a summary. The model thinks of queries, reads results, and drafts prose. The harness actually calls the search tool, sets timeouts, trims result counts, drops duplicates, decides whether to run another search or stop, and stores intermediate state in files or memory. If the harness dumps full search results into the prompt, cost explodes. If it cuts too hard, needed context disappears. Performance is the sum of model and harness.

From a systems perspective, the next question is obvious. Models get swapped as whole versions, so why does every project rebuild the harness from scratch. Teams differ, projects differ, and even tasks inside one team differ a little. One retry policy or one memory format changes results with the same model. Scattered implementations lock know how inside code, and good patterns spread slowly. That is the spot HarnessDev aims at. It proposes turning the harness from a craft object into a repeatable process.

Why hand built frameworks feel heavy now

Hand built frameworks used to be enough. Tools were few, tasks were short, and models behaved alike. A few fixed patterns covered most work. Sequential calls, simple branches, fixed retry counts. That carried a demo.

Conditions changed. More tools exist, tasks run longer across many steps, and each model has its own strengths and failure modes. One model follows tool formats well but changes plans too often. Another keeps plans steady but loses its way inside long tool output. One harness cannot cover every model. When the model changes, the team must adjust call order and memory handling along with prompts. Framework building cannot keep pace with model turnover.

What I find telling here is the clash of scale and variety. Models change within months while harnesses stay tied to human hands. Manual fixes stay precise for one or two tasks but fall behind across ten or twenty. Each task wants its own tuning, and those tunings collide. A retry policy that fits task A becomes a cost blowup in task B. Automation is the natural response. Instead of fixing each case by hand, build a machine that creates many candidates and filters them by execution.

A loop that generates runs and keeps

The process HarnessDev sketches sounds simple in words. The model creates candidate harness code, each candidate runs real agent tasks, and strong candidates survive as the start of the next generation. With each round, the harness should improve a little. Instead of drawing the blueprint, people design the selection.

Look closer at each phase. In generation, the model writes runnable harness code with parts for tool rules, planning cycles, memory reads and writes, and error handling. Early rounds favor breadth. One candidate retries with patience, another keeps memory short, a third summarizes tool output before passing it on. Variety is the raw material. If variants look alike, selection loses force.

In evaluation, each candidate harness attaches to tasks and runs. The run yields success rate, step counts, tool call counts, and recovery rates. The useful comparison keeps the model fixed and swaps the harness. That isolates the harness contribution. If both model and harness change at once, nobody learns which change helped.

In keeping, strong candidates stay and the rest go. Survivors get small modifications to form the next round. That is why the process earns the name evolution. I think the more important result is not any single score but where the selection pressure points. Pressure on success rate alone can raise success while cost explodes. Pressure on step count alone can pick hasty harnesses that rush. What gets measured and what gets kept is nearly the whole design.

A higher score does not always mean a better harness

The first place to doubt the loop is evaluation. A higher score means something only when the measuring stick stands straight. In harness evolution that stick bends easily. Once a harness fits the evaluation tasks, scores can rise while real ability stays flat.

Take a plain example. If evaluation tasks always use tools in the same order, a harness that memorizes the order scores high and then falls apart when the order shifts. If one error message appears often, a harness tuned to that message wins and then fails on new errors. If the grader forgives format slips, sloppy harnesses survive and then fail against strict production tools. The score is the compass of evolution, and a biased compass sends the ship the wrong way.

I felt some doubt at this point. When harness and task evolve together, this kind of overfit hides well. The model makes harnesses, harnesses solve tasks, and task scores feed back into harness choice. Small bias grows around that loop. A lucky strategy gets labeled as skill and becomes the start of the next round. That is why evaluation needs separate bundles. One bundle for evolution, one for midpoint checks, one kept hidden until the end. When scores on the hidden bundle move along, only then can we speak of real gain.

Reward hacking belongs in the same frame. It names shortcuts that please the grader without solving the task. Polished logs, formats shaped for the judge, skipped hard steps marked as done. Hand built harnesses show this too, but automatic selection spreads it faster. Once a shortcut gets picked, its children dominate later rounds. Unless the pipeline hunts for shortcuts, evolution will breed tricks rather than skill.

The weight of editing its own steering code

The question turns sharp because the target is the code that steers the model's own actions. This differs from ordinary code generation. The harness decides which tools the model may touch, how far access reaches, and what happens on failure. Letting the model rewrite that code puts ability and permission inside one process.

Isolation comes first. Candidate harnesses must run apart from real tools. Actions that cannot undo, such as payment, deletion, or outside transmission, stay blocked or switch to fake tools during evolution. No candidate call should reach production systems by accident. If a candidate reads outside its allowed range, reaches the network, or tries to widen permission, it should lose regardless of score. Safety checks run before score checks.

Then comes drift toward wider permission. Evolution likes scores. If wider tool access looks like it solves more tasks, selection may favor it. Scores rise now. A harness with excess permission causes a larger accident later. Open tools get misused sooner or later. So permission needs its own watch, apart from score. Each candidate should log which tools it touched and whether each touch helped solve the task. Candidates with unneeded access should not survive even with good numbers.

Records and replay matter too. Which harness got picked, which tasks it ran, which scores it earned, which edits appeared along the way, all of it should stay on file. Without that trail, nobody can walk back when trouble appears. A process that picks well but cannot explain why leaves no start point for repair. Keeping the history of choice feels tedious, yet for a system that edits its own code it reads more like duty.

What breaks when tasks and harnesses move together

The second place for care is the joint motion of tasks and harnesses. Normally tasks stand still while models adapt. In a structure like this one, harnesses move too. When both sides move, the floor of evaluation moves with them. Claims of progress turn vague.

A few traps deserve names. First, sticking to the task mix. If evolution tasks lean toward web search, search shaped harnesses win and then struggle on file cleanup or long form writing. Second, sticking to grader taste. If the automatic judge likes long answers, harnesses drift toward padding. Third, freezing around a lucky start. A structure that fits by chance in early rounds can lock in, and later edits only circle near it. A better structure farther away never gets a visit.

Known guards exist. Mix task bundles, swap judges, and now and then throw in unfamiliar tasks on purpose. A hidden bundle pulled out midstream gives a position fix. What I would watch next is exactly that midpoint curve. A chart where hidden scores climb beside training scores through the rounds would persuade me more than one final win. A clear process makes its results easier to trust.

Questions I would ask before putting it in production

A good research idea meets different questions on the way to operations. Lab scores give way to the terms of a system that runs through the night. Read this way, HarnessDev prompts direct checks. It looks good, so where would it run. What does it cost and where does it break.

Latency and cost come first. If an evolved harness calls the model many times per step and knocks on tools again and again, success may rise with the bill. Teams need per step model calls, tool calls, and token use beside success rate. A five point gain that doubles cost will not ship in many shops. Evolving under a fixed budget reads closer to reality. What was gained inside the budget is the true report card.

Memory and state handling come second. Stability shifts with where intermediate memory sits. Memory only runs fast until long tasks overflow it. Files or databases run slower but hold up. Ask which memory plan the picked harness uses. Watch long tasks in particular, how it trims memory when full and what it drops. A poor drop rule loses early clues and repeats the same error.

Tolerance of tool failure comes third. Production tools stutter every day. Timeouts, format shifts, brief outages. A good harness expects failure. It backs off between retries, routes around through other tools, and saves partial results before stopping when stuck. If evolution never injects such failure, the winner becomes a car that only drives on clear days. Wet day scores need their own runs. This is where I feel the lab to field gap most.

What a one line summary misses

Squeeze this work into one line and it becomes a model that builds its own harness. That line is not wrong, but it drops half the story. The real story sits in choice rather than creation. What gets made, what measures it, what survives, those decisions shape results. Models already generate well. People must still design the choosing.

A related hope needs a correction. Harness automation does not remove frameworks. It lifts them one layer up. Instead of hand writing each harness, people build the machine that proposes, tests, and isolates candidates. Hands move to new places rather than leaving. Task bundles, safety rules, and pressure tuning stay human work. The design may even grow harder. Writing answers by hand gives way to writing the test that picks answers.

One more belief deserves pushback, the thought that larger models remove the need for harnesses. As models grow more able, they take on more, and more scope needs more outside coordination. More tools and longer chains open more places to slip. A good driver does not replace care for the car, and a good model does not replace care for the harness. Model and harness have to improve together.

Scenes I want to see in the next experiment

What I want to see first is the hidden score curve. As evolution rounds rise, do scores on the kept hidden bundle climb with the training bundle. Joint climbs suggest generalization. A training only climb suggests memorization. That single chart would settle much of the reading of this work.

Next I want the permission and cost logs. How did tool access range shift across rounds, and how did per step calls and token use move. A gain with wider permission and larger bills is an expensive gain. A gain with steady permission and lower cost signals better structure. The mix of those three numbers says more than any lone score.

Last I want a gallery of failures. Which harnesses won with tricks and then got cut, and which safety rules caught them. A paper that shows its traps and its guards helps the field more than one that lists wins alone. For research that touches its own code, such openness reads as a condition rather than a bonus.

One thought stays with me as I close. The question HarnessDev raises is not whether to grow larger models but who makes the program outside the model and how. Treating the harness as evolving material still needs proof. The evaluation traps are real and the safety load is real. Still the direction feels right. Agent reliability is the sum of model and harness, and harnesses have rested in human hands for too long. In the next round I want the hidden curve, the permission and cost logs, and the list of caught tricks. With those three in place, the idea moves from a fine question toward an answer worth using.