When the scores match but the work does not

OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows, shared on arXiv as 2609.09203, looks at how AI scientists work instead of only what they produce. The paper argues that final outputs alone cannot tell systematic reasoning apart from lucky guesses. The original is linked under the title.

When I first opened the paper I expected another evaluation dataset. That reading did not survive for long. Three models stood side by side in the eighties on success rate, while their inner behavior looked nothing alike. That gap kept me reading.

Hello, this is Patrick. When a student brings a clean report, a teacher hands over a grade and moves on. Would that grade tell you whether the student will do well on the next experiment. I carried that question through this piece. Today I want to walk through 558 recorded trajectories with you.

When the scores match, are the skills the same

Imagine three students who all scored 85. One knew the answer from the start, one rewrote the solution ten times before getting it right, and one guessed correctly. Would you call those three performances equal. I paused here. The scores match, yet what to expect next time differs completely.

That is where this work begins. Earlier evaluations of AI scientists looked mostly at final artifacts such as code, hypotheses, or papers. How the answer was reached stayed out of the record. There was no way to separate careful method from fortunate guessing, which meant methodology could not be audited and failures could not be diagnosed with much precision.

So the authors flipped the design. They kept the full process next to the outcome and released it. Across 124 tasks in drug discovery, materials science, genomics, and literature analysis, they recorded what each model thought, which tools it called, what it observed, where it failed, and when it revised course. The number 558 did not read to me as a size boast. It read as 558 watched problem solving sessions, where grading the exam was replaced by sitting beside the solver.

What does keeping the process mean in practice. The next section opens up the shape of the record.

What remains when the working is thrown away

The relation between result and process in science is subtle. A drug candidate found through contaminated reasoning earns little trust, even when the molecule looks right. A failed experiment with clean records still feeds the next hypothesis. What every human scientist takes for granted was missing for a long time in AI evaluation.

The gap creates concrete trouble. A model writes code and the tests pass, so should we trust the code. A model proposes a hypothesis that roughly fits the literature, so was the reasoning sound. Final answers alone cannot settle such questions. The assumptions, the tool slips, and the self corrections stay invisible.

What interests me more than the headline numbers is the structure behind them. This dataset holds behavior rather than accuracy. When turning points in thought, slips in tool use, and moments of self reported confidence sit together in one record, evaluation changes character. It moves from exam scoring toward a driving log, where strengths and repairs become visible.

Recording everything does not solve every problem by itself. Sparse records resist interpretation, while dense records drown in noise. That makes the design of the record decisive. Which shape did the authors choose.

How 558 runs were gathered

The phrase 124 tasks can sound small at first. The composition changes that impression. Four areas share the load with balanced difficulty, which lets readers separate a lucky specialist from a steady generalist.

The trajectories total 558. Three frontier systems each contribute 124 trajectories with domain and difficulty held balanced, four open weight systems add 30 each, and 60 live retrieval variants sit on top. GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro meet Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B on the same task board.

I liked this layout. Frontier models alone would hide where smaller models break first, while a single domain would risk crowning a fortunate major. Varying area, difficulty, and model size together is what lets behavior stand out.

Is 558 a large number. It is not infinite, plainly. Yet these are complete trajectories rather than final answers, so the weight differs. Each one chains thousands of steps of thought, calls, and observations. I read the collection as dense rather than merely large.

The next step is to look inside a single step. What was written down that makes this analysis possible.

Why one step is split into nine slots

Anyone who has opened raw agent logs knows the feeling. There a sentence that might be thinking sits beside a tool call, while the error hides lines below. So later analysis takes real effort, and the authors blocked that mess at the source. They fixed a nine slot frame for every step, which I found clever because free form logs are easy to write but hard to read.

In that frame the current thought comes first, followed by the tool call and the returned observation. On top of that the record notes whether an error occurred, what triggered a revision, and how confident the model felt. In other words every step becomes a small lab note that says what was tried, what came back, and why the direction changed. With that layout a reader can follow where the path split without hunting across files.

What is gained by fixing the shape so tightly. Two gains stood out to me, and the first is for machines. Fixed slots make error rates, revision frequency, and gaps between confidence and performance easy to extract. The second is for people, because a reader can follow where reasoning slipped and whether the tool or the thought deserves the blame. It works like a mistake notebook kept during study.

A natural question follows here. Does asking for thoughts at every step reveal true intent. It does not fully, because self reported confidence can miss and written thoughts can rationalize after the fact. Still, asking the same questions at every step enables comparison, since departures from the shared frame become signals in their own right. So the result is less like a perfect scan and more like a photo album shot from a fixed angle.

With the shape clear, the contents come next. How did three similar looking models behave so differently inside.

Behind the same eighties on the scoreboard

Success rates for the three frontier models cluster from 84 to 89 percent. So at a glance the table suggests a tie, and I read it that way at first too. Yet once the trajectories were opened the picture changed, or more precisely differences hidden on the scoreboard showed plainly in the error logs.

The numbers behind that shift are striking. Claude Opus 4.6 left an average of 2.5 errors per trajectory, while GPT-5.4 left only 0.08. That is more than a thirtyfold gap, and the statistics look sharp with a p value below 0.0001 and a Cliff delta of 0.613. So differences hidden by success rates show plainly in error rates, which is where I stopped reading the score table.

That gap matters more than it sounds, because equal success can mean one system limped to the finish while another walked straight. Limping success travels poorly, since a changed tool or a slightly shifted dataset can break it. Straight walking success is more likely to repeat, which is why I read this scene as a driving log rather than a report card.

From a systems perspective, the next question is obvious. When error prone runs still succeed, who covered the gap. Did the agent repair itself, or did a forgiving task structure absorb the damage. This dataset supplies material for an answer, since revision triggers and retry records sit alongside outcomes. So I did not read the thirtyfold figure as a failure boast, but as a marker of success quality.

Frequency alone cannot guide a fix, though, because errors differed in character as well.

Busy hands that slip and minds that misread

Split by type, the two models show opposite faces. For Claude, 66.7 percent of errors were tool misuse such as wrong arguments, wrong function choice, or misread results. For GPT-5.4, 83.6 percent were reasoning errors where the calls looked fine while the interpretation or the assumption drifted.

People learn in similar ways. One student mistypes on the calculator, another types cleanly yet sets up the wrong equation. Equal scores call for different tutoring, with practice on tool habits for one and review of concepts for the other. Models follow the same logic. Visible failure locations decide where to intervene.

This is where I would be careful not to over-read the experiment. Frequent tool errors do not prove strong reasoning, because early tool slips reduce the chances for reasoning to be tested. A large share of reasoning errors does not prove flawless tool skill either, because fewer calls create fewer chances to slip. Share and frequency have to be read together. That is why reporting per trajectory counts beside type shares feels right.

Where does Gemini 3.1 Pro sit. It shares the success band, yet its behavioral texture differs again. Three models on one scale leave one question standing. Which scientist would we hire, the one who lands answers or the one who keeps a clean path. I lean toward the second. Science repeats the walk after repeated stumbles without losing the trail.

Time to widen the view. What shows up in smaller models and in retrieval settings.

What small models and live search reveal

The four open models each add 30 trajectories, from Qwen2.5-7B and Mistral-7B-v0.3 to Phi-3.5-mini and Qwen2.5-1.5B. These lighter systems show where stamina runs out on the same tasks.

Reading small model traces has a frank pleasure. Large models often cover mistakes with smooth recovery, so failures surface late. Small models run out of recovery strength sooner, so the stuck points appear early. Tool breaks and snapped assumptions show with unusual clarity. I read those traces as stress test records. Weaker runners sometimes make better mirrors for weak links in a system.

The 60 live retrieval variants add another layer. Live search can pull fresh papers at run time, which sounds like help, yet the process grows more tangled. Choosing queries, judging returned documents, and tying quotes to inference all enter the trace. Retrieval reads less like a master key and more like a fresh fork in the road.

A practical question follows. How would a team use this. When my colleagues and I judge a new architecture, benchmark digits alone never settle it. We ask what changed and what cost structure the change creates in operation. Small models and retrieval variants speak to that cost sense, because added points come with visible calls and revisions. For a lab with a fixed budget, that scene can help more than a success table.

With records gathered and behavior compared, use comes next. Which questions do these traces unlock.

Which problems open up once traces exist

The release pairs data with definition, because five benchmark tasks arrive together with baselines. Those baselines run from logistic regression and random forests to LSTMs and Transformer models, all aimed at judging or predicting something from traces. I was glad to see collection and use designed together, since gathering traces and putting them to work belong in one plan.

Consider the shape of the questions that follow from that design. Can middle steps foretell final success. Next, can an error step reveal whether self repair is likely. Further, does self reported confidence deserve trust. Such questions stay closed when only final answers exist, because process records open them. So I read these five tasks as reading problems rather than scoring games.

I also appreciated the modest baselines, because the tasks do not start with giant models alone. Classical classifiers and sequence models set the floor first, which shows how far simple methods reach before heavier tools enter. Readers who have seen the same model give different answers to the same question know why that matters. So I trust benchmarks with firm starting lines, because low and solid floors make progress visible.

Five tasks cannot cover everything, plainly, since questions about scientific process have no natural end. The direction still feels clear, as evaluation moves from a guessing game toward a reading game, from whether the answer matched toward how the work was done. The next section asks which habits need repair once that reading becomes possible.

When habits need repair more than scores do

A reply may come to mind here. When answers land, why mind the stumbles. For one off projects that reply holds. A single hit followed by goodbye needs no clean path.

Habits turn into cost across longer work, though. Calls multiply, retries multiply, and review load grows. The pair of 2.5 and 0.08 errors per trajectory may share a success band, yet they feel nothing alike in operation. Picture a screening pipeline running overnight. A system that cannot steady itself and rattles at every bend slows the whole line. Token spend piles up, hours leak, and locating the leak stays hard.

So I read this work as a driving habit report rather than a score table. A driver who shakes at every sharp curve needs different coaching from one who occasionally misreads a straight road. One needs tighter call discipline, the other needs stronger hypothesis checks. Diagnosis comes first because prescriptions differ.

How might such habits improve. The data holds the material. Tool slips, observations that bent a thought, and confidence misses sit step by step, ready as training and evaluation matter. What I would watch next goes beyond lower error rates toward changed error shapes. Cutting tool slips and cutting inference slips likely need different training.

One widening view remains. Which larger shift does this record point toward.

When process becomes visible, evaluation and design move

The question itself changed, from what was made toward how the work went. That change reaches past evaluation into design. Visible process shows where to touch, whether to tighten call formats, add middle checks, or adjust confidence expression. Decisions can rest on records rather than hunches.

The view matches how our team picks models. Digits alone never decide. We look at how a system fails, whether it fails quietly or loudly, whether it notices alone or needs telling, whether it repeats one mistake or moves to fresh ones. Those habits set felt cost in operation. This dataset gives a footing for moving that sense into numbers.

Some restraint fits here. Traces alone do not create rigor automatically. Written thoughts remain model prose, and confidence scores remain self grades. Weight arrives when they join harder records such as calls and observations. I see the release as a fine microscope rather than a truth machine, a tool that shines brightest for readers who know what to look for.

What remains for people then. I think human eyes grow more valuable as process turns transparent. Reading records, naming what feels off, choosing the next task layout, and catching smells that digits miss all stay human work. The 558 trajectories are homework books rather than answer sheets. How we read those books will shape the next level of skill.

The traces, the schema, the harness, and the task definitions are shared under CC BY 4.0, and the original is linked under the title. I expect to keep this material on my desk for a long time, and to reach for it whenever I want to see how work was done rather than only whether it scored.