Bringing agentic evaluation into one harbor: Harbor Adapters and Harbor-Index

Patrick Rho · AI Research

The paper covered here is Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation, released on arXiv as arXiv:2609.04298. The original is linked under the title.

The claim is that porting more than eighty agentic benchmarks onto one unified infrastructure, comparing eight models at large scale on top of it, and then distilling the pool down to the eighty two task Harbor-Index yields a demanding standard on which no model and harness combination passes thirty percent, and this post walks through why agentic evaluation stayed fragmented, what the adapters unify, what the large comparison revealed, and why the selected eighty two tasks matter.

Why agentic evaluation stayed fragmented

The number of agentic benchmarks has grown fast. There are collections that measure coding assistance, collections that measure web navigation, and separate collections for tool use, long horizon planning, and multistep reasoning. By count alone, the field looks abundant. Yet for a team that wants to run one agent across many benchmarks, abundance reads less like wealth and more like fragmentation. Each collection brings its own runtime environment, its own way of connecting agents to tasks, and its own scoring procedure. Every new agent must knock on each benchmark door separately. Installing environment dependencies, fitting harness joints, and matching scorer versions repeats with each stop.

The fragmentation has roots in the nature of agentic tasks. A short answer question for a language model can travel over a single line of text. An agentic task needs an environment. It may need a terminal, a browser, a file system, a code repository, or a test runner. The way environments get launched differs by benchmark. Some arrive packed in containers, some demand cloud sandboxes, and some lean on specific versions of specific tools. Because the environment is part of the task itself, evaluation cannot unify unless environments unify. Each benchmark growing into its own environment was natural, but the result leaves evaluators commuting between islands.

A second split sits at the agent connection point. Some benchmarks bind tightly to one particular agent scaffold. Solving the task means using the harness the benchmark assumed. A harness is the frame that governs how a model interacts with an environment. It fixes the format for tool calls, the way observations return to the model, and the number of permitted attempts. When benchmark and harness form one body, model capability and harness convenience blend together. When a score rises, it stays unclear whether the model grew smarter or the harness grew kinder. The fairness of comparison wobbles at exactly that joint.

So evaluations in practice often end in compromise. Teams run a handful of representative benchmarks, accept scores on each default harness, or skip tasks whose environments resist setup. Numbers come out, but interpretation stays thin. It remains unknown whether an agent handles a broad spread of work or only shines on a few chosen collections. The difficulty this paper names at the outset is exactly that one. Evaluating agents across the growing set of agentic benchmarks keeps getting harder because complex environments and agent integration demands pile up. Many practitioners will nod at that diagnosis. Benchmarks enjoy a harvest while evaluation suffers a famine.

What Harbor Adapters unifies

The first contribution of the paper is Harbor Adapters, a unified evaluation infrastructure. The harbor metaphor reads naturally. Differently shaped vessels need standardized docking equipment before they can share one port. Harbor Adapters ports more than eighty agentic benchmarks so that arbitrary agents can be evaluated on them. The word porting deserves a clear definition. It does not mean copying benchmarks. It means relocating them onto a shared evaluation frame while preserving what the original tasks meant. Environments and scoring stay intact while the connection point to agents follows one common specification.

The unification matters because it changes the unit of evaluation. The unit used to be the benchmark. Reports took the form of scores on this or that collection. With adapters in place, the unit becomes the agent. One agent can be set down on many benchmarks and its record read straight across. The research question changes with the unit. It moves from who leads on this benchmark to how far this agent travels across a broad task distribution. The field of view widens from a single task to a task ecosystem.

The phrase arbitrary agents deserves attention too. It signals that agents tied to no particular scaffold can run in the same frame. An experimental agent built by a research team and an agent built on a commercial harness can both dock at the same equipment. Comparisons widen once that holds. Models can be compared, harnesses can be compared, and model plus harness combinations can be compared. Infrastructure opens up possibilities in that literal sense. When setting up environments and refitting joints stops consuming the calendar, the freed time converts into more comparisons run.

Naturally, such unification invites suspicion. Squeezing different benchmarks into one frame might wash out what made each task distinctive, since wrapping a browser state for web navigation and a test runner for coding in the same interface could blur fine meanings. The answer this paper gives to that suspicion is the validation procedure covered in the next section. It does not merely declare the ports done. It pairs the declaration with evidence that the ported results stay faithful to the originals. The stance on display is that the quality of unification comes before its announcement.

How to trust the ports

Porting more than eighty benchmarks is a statement of quantity. In evaluation infrastructure, quantity must wait behind quality. The first questions concern meaning. Do the relocated tasks measure the same thing as before. Do scoring rules operate as originally intended. Do environment differences seep into scores. Without answers to those questions, unified numbers stand on sand. The paper does not hurry past this point. It reports that the adapted benchmarks were validated through rigorous code review and parity experiments to preserve fidelity to the originals.

Code review unpacks into a concrete practice. Humans reread the code written during relocation. They check whether environment setups mirror the original task conditions, whether scoring logic follows the original rules, and whether the observations and tools handed to agents stay inside the original scope. Automated tests alone cannot catch drifts in meaning. A reviewer eye catches the rest. Mistakes like interpreting time limits differently per environment, widening the permitted tool list, or leaving open a shortcut the original forbade are the kind that reading eyes catch well. The qualifier rigorous signals substantive review rather than a ceremonial pass.

Parity experiments form the second stage. They check experimentally whether runs on relocated tasks point the same way as runs on the originals. The question is not byte identical numbers, since environments and run conditions must differ slightly. The question is closer to preserved ordering and matching tendency. Tasks that were hard in the original should stay hard after relocation, tasks that were easy should stay easy, and a strong baseline should keep leading a weak one. Once that parity holds, scores on adapters can be read in the same language as scores on originals. The digits may shift, but what the digits mean carries over.

That validation posture matters because trust in infrastructure accumulates through procedure. One or two relocated benchmarks can be watched by a researcher eye directly. Past eighty, individual attention stops scaling. Procedure must carry the trust instead. Pairing a human eye with a numerical eye is persuasive for that reason. Reviewers catch drifts in meaning while experiments catch systematic skew. Two eyes catch two different kinds of error. Reading this paper, I felt reassured at exactly this point. The quantity claim arrived with a quality procedure attached.

A large comparison across eight models and fifty four benchmarks

With infrastructure ready, the next move is to use it. The paper runs a large scale study that evaluates eight models across fifty four benchmarks. Each number deserves unpacking. The eight models reportedly span capability tiers. The lineup stretches from strong frontier models down to lighter ones, so representatives stand at each level rather than one crowded band holding a private contest. Spreading tiers changes the texture of inquiry. Discussion moves past who takes first place toward where the tiers part company across the task distribution.

The fifty four benchmark scale comes next. It reads as the subset of the eighty plus ports actually used in the large comparison. The scale moves past the accident of any single collection. One bundle can flatter by luck and another can punish by luck. Fifty four bundles leave less room for luck. Strengths and weaknesses of an agent surface as the shape of a distribution. One tier travels evenly wide while another spikes only in its home territory. The purpose of evaluation shifts at that point from ranking to mapmaking.

The mere feasibility of the comparison proves something about the infrastructure. Had fifty four different environments needed hand assembly, the study could hardly have finished inside a research calendar. With adapters, setting the same agent down at port after port becomes procedure. Researchers spend their hours on interpreting results instead of installing environments. Infrastructure papers often fall into a familiar trap. They build the bridge and never show traffic crossing it. This paper avoids the trap. It builds the infrastructure and then runs heavy traffic across it. The crossing itself becomes the user manual.

One boundary belongs alongside the praise. Fifty four sounds large, yet it cannot contain every agentic capability. The distribution of ported benchmarks is the horizon of vision, and whatever lies past the horizon stays unmeasured. If brand new interaction styles or very long horizon tasks appear in none of the existing collections, those abilities stay off this map. Even so, a fifty four wide view reads far more honestly than a single benchmark report. It offers a chart of an archipelago instead of a portrait of one island.

Separating model capability from harness effects

The most instructive part of the large study is its design. Every model runs under Terminus-2, a shared harness, and also under one of three native harnesses. The intent is to separate model capability from harness effects. As noted above, an agent score is a joint product of model and harness. The same model scores differently when the harness changes, because tool formats, observation volumes, and attempt budgets all shift. A single condition score therefore reads blurry. When credit looks good, it stays unclear how much belongs to the model and how much to the harness.

The shared harness condition clears that blur. Placing every model on Terminus-2 fixes the harness side. Score gaps then read closer to the model side. None of this isolates some perfectly pure model essence, since a shared harness may still favor one model style over another. Still, the footing turns level at minimum. Gaps on level footing deserve interpretation. Which model travels farther can be asked without blaming the ground.

The native harness condition supplies the opposite information. Running each model with one of three native harnesses approximates how the model looks in the field. A harness tuned by a model team for its own model is built to draw out that model strengths. Scores there sit near the realistic ceiling of the combination. Laying shared harness scores beside native harness scores tells two stories at once. One story concerns the model itself, and the other concerns the combination. The design reads honest in giving two numbers to two audiences. Research teams designing the next model need the unadorned figure with harness makeup removed. Adoption decisions in practice need the combination figure, since what ships is never a bare model but a bundle.

That dual condition design carries practical weight because buying and deployment choices happen at the bundle level. Nobody in the field adopts a model alone. They adopt a model plus harness bundle, and they compare felt experience across products. That felt experience tracks the combination score more closely. This study serves both readers without forcing one number to answer two kinds of questions.

Harbor-Index as a selected set of eighty two tasks

After infrastructure and the large comparison comes the second contribution. Harbor-Index distills the adapted task pool into eighty two difficult, diverse, and high quality tasks drawn from twenty nine benchmarks. The numbers repay attention. Eighty two tasks fit inside a single day run while still spanning twenty nine sources. Teams that find a full fifty four benchmark sweep heavy need a light representative set. A representative set must satisfy two demands at once. It should stay light without losing the flavor of the whole. Eighty two items with twenty nine sources aims at exactly that balance.

Difficulty forms the first axis. Overly easy tasks carry no signal. Crowded perfect scores cannot separate contenders, so the filter keeps hard tasks. Diversity forms the second axis. A single genre collection makes a biased yardstick. Eighty two coding tasks would measure coding, and eighty two navigation tasks would measure navigation. Spanning twenty nine benchmarks keeps the yardstick honest. Quality forms the third axis. Tasks with vague prompts, unstable scoring, or fragile environments cannot serve as measures no matter how hard they look. The triple standard of difficulty plus diversity plus quality defines the selection.

The phrase meta-dataset also needs a gloss. It means curated rather than created. The tasks come from existing benchmarks instead of fresh invention. Curation over creation carries clear advantages. Each task arrives with a tested source and scoring method plus its original context. Selecting good existing tasks beats writing eighty two new ones from scratch for quality control. The drawback mirrors the advantage. Selection cannot escape the distribution of existing tasks. Entirely new abilities with no existing coverage cannot enter the set. Yet as an evaluation standard, curation is the reasonable choice. It builds a hard exam from proven materials.

The intended use of the selection is crisp. It separates strong models from each other. On an easy exam, the top band crowds near full marks and ordering collapses. On a hard exam, scores spread out. Eighty two hard tasks become the instrument that reveals gaps between leading models. The curation procedure in the next section and the score distribution after it support exactly that use.

The curation pipeline of filtering, audit, and fix loops

How the eighty two tasks got chosen is where the credibility of the paper can be weighed. The adapted suite serves as the starting pool, refined through difficulty filtering, machine and human audit, and an audit and fix loop. Each stage unpacks simply. Difficulty filtering removes easy tasks. The score distribution from the large comparison shows which tasks discriminate. Tasks that everyone answers correctly leave the exam. Tasks that nobody answers correctly need thought too, since they may measure impossibility rather than discrimination. What remains carries a live texture of discrimination. Strong models often miss these tasks yet occasionally solve them.

Machine audit and human audit form a double lock on quality. Machine audit sweeps fast and wide. It can comb large volumes for vague prompts, scoring loopholes, and environment instability. But machine eyes bring machine bias. They may flag certain phrasings while underrating certain failures. Human eyes stand on the opposite side. They read slowly but grasp context. They judge whether a task intent stays alive and whether scoring rules stay true to that intent. Using both eyes repeats the philosophy seen in the adapter validation. Different eyes catch different kinds of error, so the procedure hires both.

The audit and fix loop is the finest part of the procedure. When audit finds defects, the task is not simply discarded but repaired and reaudited. Pure discarding would make selection easy. Saving good tasks is the hard part. Rewording one sentence can clear ambiguity, and correcting one scoring condition can lift stability. Repeating repair and reaudit converts tasks that nearly got discarded into tasks worthy of the exam. The presence of the loop means curation is not mere thinning but finishing. It covers sorting rough stones plus cutting the selected ones.

Seen from practice, the procedure resembles an exam board. Good exam questions rarely emerge in one draft. They pass through drafting, review, revision, and re-review. The Harbor-Index curation runs that cycle transparently across eighty two items. Transparent procedure breeds trust in results. When scores come out low, blaming the tasks grows harder. Readers look at model limits before suspecting task difficulty. That is how a hard yardstick earns authority. An honest making process produces honestly read measurements.

What the thirty percent wall says

Now to the scores. No evaluated model and harness configuration exceeds a thirty percent pass rate on Harbor-Index, with the strongest configuration reaching 28.0 percent with GPT-5.5 with Codex. How should those numbers be read. From the yardstick view, they promise ample room for discrimination. An exam near full marks cannot separate leaders. Everyone gathered below thirty percent means the ceiling sits far above. Future models have long climbs available on this exam, which gives the yardstick a long working life.

The numbers also give an honest reading of current agent levels. The eighty two tasks come from existing benchmarks. They are not freshly invented impossibilities. Each task looked solvable inside its home benchmark. Yet gathering only the hard and diverse ones leaves even the strongest combination short of thirty percent. The gap speaks loudly. Scoring high on one benchmark and performing evenly across a broad distribution are different achievements. The champion of one island struggles across the archipelago. The signal points past narrow optimization toward broad generalization.

The makeup of the top score deserves notice too. The peak pairs a strong model with a well matched harness. Both the model tier and the harness tuning matter. The reading connects with the dual condition design discussed earlier. If shared harness scores show the bare face of a model, the top combination score shows something near full field strength. That even full strength falls short of thirty percent proves the difficulty of the exam. It also reminds builders that pairing matters. Placing a good model on a good harness moves scores in substance, not just style.

None of this needs a pessimistic reading. A benchmark reading proves more productive. The thirty percent wall is a survey marker of the current position. Markers exist so movement can be measured. Watching where models climb on this exam over the coming year will show agentic progress at a glance. The instrument filters out the score inflation of individual benchmarks and isolates genuine advance. The virtue of a hard exam lies not in low scores today but in meaningful score growth tomorrow.

What the open releases make possible

The authors release the adapters, evaluation results, in depth analysis, and Harbor-Index as open source artifacts. Each item in the list opens something specific. Releasing adapters means other teams can evaluate inside the same frame. Nobody needs to reinstall environments from zero or step on the same mines the porting process already cleared. Once a new agent connects to the docking equipment, more than eighty benchmarks become its home port. Lower entry barriers multiply comparison counts, and multiplied comparisons sharpen the field map.

Releasing evaluation results matters just as much. The score tables across eight models and fifty four benchmarks are data in their own right. Other researchers can run their own models in the same frame and set them beside these tables. Past scores become baselines without remeasurement. Releasing in depth analysis shares the starting points of interpretation. Observations about where tiers part company and where harness effects loom large save followers from repeating the same excavation. Shared interpretation moves the conversation to the next question sooner.

Releasing Harbor-Index shares a light yardstick. Eighty two tasks keep compute costs modest. Each new model release can take a quick turn on it. The paper claims arrive in reproducible form that hands can grasp. The thirty percent wall stops being a numerical declaration and becomes an exam anyone can run. Objections can be settled on the same exam. Disputes move from words to runs, and openness raises the quality of dispute.

Openness is not all sunshine. A public exam invites fitting to the exam. Training against the eighty two tasks until scores rise without abilities following is the familiar shadow behind every published yardstick. Still, the gains from openness outweigh the shade. The overfitting risk can be managed operationally through periodic refreshes. A private yardstick may block overfitting while slowing the field. When everyone can measure with the same ruler, the field moves fast. Choosing release reads as choosing speed.

What remains and what to ask next

The boundaries of this work belong in the record too. The distribution of ported benchmarks is the horizon of vision, as noted before. Abilities past that horizon stay unmeasured. Parity experiments support fidelity, yet they cannot promise identity with the originals. Fine environment differences may still favor one agent style. The dual condition design tries to isolate harness effects, yet the shared Terminus-2 harness keeps its own bias. Every yardstick retains the shape of its maker.

What to watch next is equally clear. First comes the refresh cycle of the selected set. Eighty two tasks look hard today, but climbing models will soften them eventually. Extending yardstick life needs periodic reselection. The loop of difficulty filtering, audit, and repair should turn again to distill a fresh eighty two. Second comes distribution growth. As new interaction styles appear beyond existing benchmarks, porting targets must widen. The eighty plus count should remain a growing port rather than a fixed wall. Third comes deeper analysis of harness effects. Learning which kinds of harness help which kinds of tasks would guide agent design directly. The division of labor between scaling models and refining harnesses would come into focus.

A short sentence remains at the end. Measuring agents needs a harbor, a harbor yields a map, and a map shows the road ahead. This paper delivers docking equipment plus a pocket chart of the archipelago together. It marks a survey point where even the strongest combination stops below thirty percent, and it leaves that marker open for anyone to use. Whether evaluation stays a story of scattered islands or becomes traffic through one shared port now belongs to the users. As measuring with the same ruler and competing on the same exam accumulate, progress in agents will read as numbers rather than impressions.