Reading Repo-To-Skill: what agents lack is not models but operational knowledge
Following the DisCo paper on distilling operational knowledge into skills, with a close look at the AREX-Skill library and what the matched-budget experiments actually show.
Read the original paper
Patrick Rho · AI Research
This post is a close reading of the arXiv paper "Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills". I have linked the original paper right under the title, so if the summary pulls you in, open it side by side.
The claim fits in one sentence: if a machine can read a well-kept GitHub repository and distill it into reusable "skills", an agent goes further on the same budget. The paper starts from a diagnosis that the biggest bottleneck for agents doing real research is not model quality — it is the operational knowledge of which package to use and how to configure it sitting outside the system. In this post I walk through where that claim comes from, how the DisCo distillation framework works, and what the experiments actually show.
Whenever I hear that agents now carry out machine-learning research end to end, I ask the same question. Models keep improving and harnesses keep getting more sophisticated, but where does the know-how live, the kind of thing any practitioner takes for granted? Which package to use, how to configure it, where runs usually fail.
The Repo-To-Skill paper takes that question head on. It names the missing layer, then shows how to build it automatically. The missing layer is operational knowledge, and the framework is DisCo, which distills that knowledge into skills.
The empty slot between model and harness
Agent systems are usually described with two components. The model supplies understanding, planning, and execution. The harness supplies orchestration, memory, verification, and refinement. Each has improved on its own. Models get stronger every generation, and harness engineering keeps maturing.
Neither supplies operational knowledge. The model's prior is broad but fixed, and the harness controls procedure without providing domain content. The paper goes as far as writing this down formally. A research agent should be the model plus the harness plus an explicit operating context K.
I like this framing because the most expensive part of research practice is rarely the elegant idea. It is boring trial and error. Inferring package behavior by running it, spending budget on misconfigured runs, then starting over from scratch on the next task because nothing was retained. Naming where that cost structurally belongs already makes the paper worth reading.
Distillation comes in two directions
DisCo distills in two forms depending on where the run is anchored. One is task-agnostic: condense widely used repositories and everyday tools into reusable skills ahead of time, available to any later task. The other is task-oriented: decompose a concrete task, find the capabilities the agent cannot already supply, and produce the skills that fill the gap.
Either way the process has the same four stages. Decide which capabilities matter, gather the evidence, assemble a skill graph, then verify. Verification is the point. No skill is admitted on the strength of its sources alone, and surviving gaps are recorded rather than hidden. That reads like a declaration that summarization is not distillation.
The skill as a container matters
The paper's container for operational knowledge is the skill, in three layers. SKILL.md is the knowledge interface, references carry the substrate, and scripts form the execution interface. Only the entry layer is read up front. The rest unfolds on demand.
I find this choice practical rather than fashionable. Agent systems already understand this format, so neither the model nor the harness has to change. An agent can hold thousands of skills yet read only the few a task needs, which keeps the context budget manageable. The format was chosen because it satisfies the conditions for commandable knowledge, not because it looks elegant.
One source rarely fits in one skill, so the output is organized as a graph. An entry skill states the scope and routes to component skills. The agent follows the router and opens only the branch its problem calls for.
The scale of the library, and its router
Run the ahead-of-time form across the ecosystem and you get the AREX-Skill Library. It distills 1,000 repositories into 5,353 verified skills, organized into 20 areas and 178 capability families with a router on top. Reported construction cost is about $40 per repository.
The router was not thrown together either. Repository summaries are frozen first, the taxonomy is fixed before any assignment, and each placement requires evidence plus a confidence value. Keyword-only or dependency-only matches are rejected. That is the approach of people who understand that retrieval precision decides whether the whole thing works.
Results under matched budgets
Now the numbers. The evaluation fixes the Codex harness and the GPT-5.5 backbone and varies only the skills. Construction budget is kept separate from execution budget. Remember that condition, because it flatters the skills side slightly. Total cost is higher with skills included.
On the full 75-task MLE-bench suite, the Any-Medal score rises from 31.11% to 72.89%. That is 41.78 points, or 134.3% in relative terms. The lift is largest on hard tasks. The High split goes from 13.33% to 62.22%, a 4.67x improvement.
I read this pattern as follows. Easy tasks can be solved bare-handed. Hard tasks expose the agent to a wide space of libraries and implementation choices where getting lost is expensive. Skills move the agent into a productive region sooner.
On 20 PaperBench replication tasks the average rises from 29.45% to 39.59%. Skills help on 18 tasks and hurt on 2. The two regressions interest me more than the average. Retrieved skills can distract from a strategy the base agent would otherwise find on its own. That is a retrieval-precision tradeoff, stated honestly, and it keeps the paper from reading like an advertisement.
It is not the extra tokens doing the work
On 188 FrontierCS tasks the score moves from 70.63 to 77.14, a 9.2% gain. The detail I spent the most time on is the usage relationship. Skill runs consume more tokens and steps, yet per-task gains are essentially uncorrelated with the additional usage. Spending more is not what produces the improvement.
The comparison line is fun too. Under the same setup, Codex with skills uses less than a third of the tokens of the Claude Code configurations while scoring higher. That is a genuine efficiency-frontier result.
On PassNet the aggregate score rises from 1.343 to 1.5313, and failed samples drop from 14 to 5. Correctness moves from 81.35% to 90.76%. On a graph-compiler task where passes must preserve semantics, fewer failures means the procedures, rejection criteria, and recovery steps in the skill graph are doing real work.
Pay once, use many times
The most practical part of this design is its cost asymmetry. The creator side pays once per source. The researcher side pays only for what a task actually opens. The library expands offline at whatever breadth the ecosystem allows, and tasks inherit the result without re-deriving it.
That is why the scaling argument holds together. Still, 1,000 repositories at roughly $40 each is a $40,000 library. Expansion is not free. But once built, every additional task amortizes the cost further, so unit economics improve with usage rather than degrading.
Where I would stay cautious
Liking the paper does not mean accepting all of it at face value.
First, every evaluation uses Codex as the harness. Whether the same library transfers to other execution environments is unshown. If it does not, the operational-knowledge-layer claim shrinks toward a Codex prompt collection.
Second, construction budget sits outside the run-time comparison. That separation is fair, but from a total-cost view the skills condition spends more. With few tasks the arithmetic could flip.
Third, repositories drift. Releases change behavior and fill in missing pitfalls. How long a distilled skill stays valid, and who re-verifies 5,353 of them on what schedule, remains open.
Fourth, the two PaperBench regressions make routing precision a live variable. Opening an irrelevant skill can actively hurt. The system needs the judgment to skip routing when nothing fits, or a fallback to unguided reasoning.
What I want to see next
Scale comes first. A much larger backbone changes the relative cost of reconstructing local patterns. Skills could matter more, or the backbone could handle that work so easily that the relative gain shrinks.
Harness generality is next. I want the same library tested under a different execution environment. Without that, the generality claim is a hypothesis.
Maintenance needs an answer too. When repositories change, skill graphs must be re-verified. At this scale I want to see the cadence and who pays for it.
Other ecosystems would be informative as well. The router and taxonomy were built over English-centric repositories. Whether distillation works at the same quality elsewhere is a separate question.
One thought that stays
Research-agent discussions usually go toward bigger models or better harnesses. Both matter.
What this paper leaves me with is a third slot. The knowledge a task needs that lives in neither the model nor the harness: what the agent knows when the work begins. Packaged as skills, admitted only with verification, built once and used many times.
The next time I evaluate an agent system, I will add a third question next to backbone and harness. What does this agent know when it starts working, and where did that knowledge come from, and how far should I trust it?