When decisions move, labels disappear and the grounding audit stays open

Decision Shifts, Lost Label Functionality, and an Inconclusive Grounding Audit in Correctness-Gated Multi-Teacher Distillation appeared on arXiv as 2609.09702. The site shows a link to the original paper under the title where the full text can be read.

The paper asks how correctness weighted multi-teacher distillation moves candidate decision accuracy separately from rationale grounding.

Hello, this paper is easy to misread from numbers alone

Hello, I am Patrick.

When several people work together, we decide whose words deserve more weight. We listen more closely to someone with a record of correct calls and discount someone who missed often. So what happens when several AI teachers train one student in the same way.

I stayed with this paper because that question feels obvious until the measurements arrive. Giving more weight to teachers who chose correctly sounds like it should produce a smarter student. Yet while I followed the runs, the averages looked better and something still felt off. The match rate rose while the vocabulary of labels shrank, and the claim about better grounding wobbled once I looked at how the audit was drawn. That unease is worth unpacking slowly.

In this post I will trace how decisions moved, which label behavior was lost, and why the grounding audit could not close. I will keep decisions and grounding on separate scales the whole way.

So why are decisions and grounding different problems

Let me pose a plain question. Is choosing correctly the same skill as giving reasons that stay tied to sources.

A school exam offers a useful parallel. One student circles the correct choice while another writes a sound derivation. A test that only checks the choice rewards both students equally. A test that reads the derivation penalizes the first student even when the choice matches. The paper draws a similar line. Whether a candidate decision is correct and whether the rationale stays tied to evidence are separate objectives.

That separation earns its keep in distillation work, because the student learns answers and explanations together from teachers. It is tempting to assume that better answers imply better explanations. But suppose a teacher adds a smooth explanation that slips in material with no source behind it. The student can then learn two habits at once, choosing well and adding unsupported texture. I paused longest on this distinction, because it decides how every later number should be read. Before asking how large a gain is, we should ask what was measured.

With that split in mind, later results become harder to misread. A rise in accuracy and a firmer tie to sources belong in different sentences. As we will see, the weighted arm leads on accuracy while the grounding audit stays open. Placed in one basket, those outcomes could be spun either way. The paper keeps them apart, and I will do the same.

The test bench was fixed before teaching styles changed

Before comparing numbers, I want to look at the bench. When benches differ, comparisons entertain more than they inform.

The study placed eight arms on shared ground. The shared ground covers 4,330 sources, one student with 63.9M parameters, 12,990 optimization rows, 406 updates, plus common evidence inputs and a common decoder. Seven teacher based arms drew from one fixed pool of three responses per case. In other words, the study held the material, the student size, and the step count steady while varying how teacher signals were handled.

I find that design reassuring. When my team looks at a new AI architecture, we seldom stare at benchmark scores alone. We ask what actually changed and what cost structure that change creates in a system. Seen that way, this setup reads cleanly. The study did not add more material or enlarge the student. It changed the handling of teacher signals.

The comparison spans four broad styles. Plain distillation learns teacher outputs as they arrive. Hard filtering removes wrong teacher signals before learning. Correctness weighting turns the volume up or down by whether the teacher chose correctly. A separate arm learns directly from source labels through supervised fine tuning. That layout invites a natural question. Does weighting add anything beyond filtering. So how did the numbers move.

Weights moved the numbers yet the joy was short

Against unfiltered distillation, the correctness weighted arm clearly moved. Accuracy rose by 0.1660, five label macro-F1 rose by 0.1323, and the conditional unsafe action rate fell by 0.4979. On shared ground, the direction looks favorable. My first pass nodded along here.

Trouble starts in the next step. Movement in averages does not imply evenly better behavior. Accuracy counts how often choices match. Macro-F1 watches whether rare labels are handled across the set. The conditional unsafe rate watches how often risky choices appear within a specific slice. Three scales can point in different directions, and here the details under the averages feel uneven. Behavior by label did not move together.

A classroom parallel helps. Imagine a report card where the average climbs while one subject drops to zero. The average suggests stronger study, while the subject scores invite hesitation. The same pattern shows up here. Averages rose while Refuted recall sank to the floor, as the next section describes. The paper marks this shift as not uniformly better, and I adjusted my reading habit at that line. Averages deserve a glance, while distributions deserve a longer look.

The model learned to answer while losing words to answer with

This is where the story turns uncomfortable. The correctness weighted arm recorded zero Refuted recall in every seed. That means it never labeled as Refuted a case that called for Refuted. In two seeds the pattern went further. The arm labeled all 167 claim examples as NotEnoughInfo, repeating the abstention label across the whole slice.

Why would louder weight on correct teachers produce that shape. Listening harder to frequent correct answers can teach the student to forget rare answers. Refuted is infrequent yet needed at precise moments, and training can remove the reason to utter it. It resembles a meeting where majority views dominate until minority views stop being voiced.

I hesitated here because the metric itself changes meaning when a label empties. Five label macro-F1 tries to reward even handling across labels, and an empty label bends that purpose. A higher average paired with a smaller vocabulary hardly reads as stronger learning. The phrase lost label functionality felt apt once I saw the split. Pressure toward one region of decisions cost the model part of its speech.

So is this a minor side effect. I do not read it that way. In document adjudication, silence on Refuted carries operational weight. When a false claim needs a direct rejection and the model only abstains, decisions stall and queues pass the work to people. Response slows as review piles grow. That hidden queue behind the accuracy number struck me as the most sobering point in the run.

The broadest matcher was the simplest trainer

A wider view sharpens the picture. The arm trained directly from source labels reached mean macro-F1 of 0.586, the highest among the arms. Learning labels straight from sources preserved broader classification behavior than learning through weighted teacher talk. A shorter path lost less along the way.

Hard filtering deserves attention too. It already reached 0.660 accuracy with 0.530 macro-F1 and 0.135 conditional unsafe rate. Removing wrong teacher signals before training built a firm floor. Against that floor, the weighted variant showed no demonstrated incremental decision gain in this implementation. Asked whether added weighting puts something above filtering, the honest reply stays close to not yet shown.

That leaves a practical question. Where would this run in production. Filtering is easy to explain and easy to operate, because the story stays legible. Bad material is removed, then learning proceeds. Weighting adds dials and more intricate behavior. Without a clear premium in decision quality, why accept the added intricacy. I came away treating filtering as the baseline. Any new weight should first answer whether it clears that bar.

To hold the numbers together before the next figure, the shape looks like this. Weighting improved accuracy and macro-F1 over no handling while lowering the unsafe rate. It still fell short of the breadth kept by source label training, and it has not shown that it rises above the floor built by hard filtering. Rising averages beside shrinking function is the summary I will carry forward.

Why the grounding audit ended without an answer

Now the rationale side deserves its own hearing. If decisions moved, did grounding improve. The audit tried to answer, and the answer stayed open.

Read alone, the counts look like this. Evidence supported positives were 0 of 20 for the weighted outputs against 1 of 20 for unfiltered outputs. Positives containing unsupported material were 20 of 20 for weighted against 19 of 20 for unfiltered. Both arms look thin on grounding, yet the gap hardly supports ranking. The sample is small and the draw has flaws.

I read slowly here because the samples were not paired. The audit did not place two outputs from shared sources side by side. It drew separate samples from mismatched source sets. Sources differ in difficulty and in how easily supporting text can be found, so a gap of 0 against 1 could reflect material rather than method. Overlap across sources was not tracked with serials either. Without a map of where sources overlap and where they diverge, a common source grounding effect cannot be estimated.

Seen that way, even 20 of 20 needs care. It sounds like everything failed, yet the scale itself was unsteady because the two arms were never placed on the same pan. The paper calls the audit inconclusive, and that label felt fair. Neither improvement nor harm has been shown on grounding.

One change in order changes what you measure

The audit story continues into ordering. In this pipeline an availability amendment to sources arrived after automatic summarization and before annotation. A mid pipeline edit entered after the material had already been reshaped.

Consider the mechanics. Automatic summarization first shortens and reshapes the source. A source amendment after that step compares states that no longer share a baseline. Annotation then views edited material, so the annotator never sees grounding in the original shape. To speak about better or worse grounding, measurement needs shared sources. That shared denominator is missing.

A nightly operations scene makes the point concrete. Suppose disclosures arrive late, the summarizer runs first, and source fixes land in the morning. Once the summary hardens, gaps in grounding become hard to trace. The morning review line must keep moving, yet the team cannot tell which stage introduced the gap. I weighed this ordering issue more heavily than lab scores might suggest. Morning operations depend on stages whose sequence stays stable.

So should this audit be discarded. I would keep it as a marker of what was not measured. Saying that a common source effect was not estimated, and that system level gain or harm cannot be claimed, gives the next design a clear signpost. Pair outputs by shared source next time, track overlap with serials, widen beyond 20 cases, and freeze the amendment point. An inconclusive tag reads less as failure and more as a door toward better measurement.

Viewed as a live system the next questions arrive fast

After the numbers, system questions remain. What would change if this approach joined a live document adjudication line.

The first thought concerns label distributions. Batch averages for accuracy appear quickly, while rare labels like Refuted move quietly unless watched. A seed that marks all 167 claims as NotEnoughInfo would surface in operations as delay. Human review queues would grow and replies would lag. I would therefore place per label recall near the front of the dashboard. The habit to build is watching distributions rather than averages alone.

The second thought concerns cost structure. A fixed three response teacher pool keeps research tidy, while teacher calls in production add spend and latency. A 63.9M parameter student can serve lightly, yet added handling during training raises experiment management cost. When hard filtering already builds 0.660 accuracy with 0.135 unsafe rate, the premium from weighting needs explicit arithmetic. Without a visible premium, the simpler path is rational in operations.

A third thought concerns display. With supported positives at 0 of 20 against 1 of 20, and unsupported material at 20 of 20 against 19 of 20, what should users see beside a decision. Shipping decisions while deferring rationale can mislead when grounding is thin. The early split between decisions and grounding returns here with practical force. I would score and show decisions apart from grounding. Separate scales read more honestly than a blended score.

What I want next is a fairer comparison

To close, I will gather the open threads. Where could this line grow firmer. That story belongs to future runs, and I will revisit it when they land.

First I want a paired grounding comparison on shared sources. Since the current audit drew unpaired samples and stayed open, the next step should place two outputs from the same source side by side and score support there. Track overlap with serials and move beyond 20 cases. Before debating 0 against 1, fix the scale itself.

Second I want a test that separates weighting from filtering. Hard filtering already holds 0.660 accuracy with 0.530 macro-F1 and 0.135 unsafe rate, so the added value of weighting should be isolated above that floor. Vary weight strength and pool composition while watching when Refuted recall breaks. Learning the conditions that push 167 claims into uniform abstention would already help operations a lot.

Third I want to understand how to keep the breadth held by source label training. The mean macro-F1 of 0.586 stayed with me less as a score and more as a shape. Short paths preserved breadth while longer interpretive paths narrowed speech. Could a guardrail on label distribution sit beside weighting, keeping breadth while using teacher signals. That combination feels worth trying.

The clearest line this paper leaves is simple to state. Whether a candidate decision is correct and whether its rationale stays tied to sources are different objectives. Refusing to call a decision shift an across the board gain, naming the vanished label, and calling the audit inconclusive rather than favorable are stances I will remember. The next time I touch a document adjudication system, I will ask for distributions behind averages and place grounding on a separate scale from decisions. How does your team read the pair today, averages first or distributions first.