Whose doubt do you trust when everyone says yes, CUSP and collective uncertainty

The paper CUSP: Decomposable Collective Uncertainty for Multi-Agent Multimodal Reasoning was released as arXiv 2609.05708. The original is linked under the title. It proposes a way to measure system level uncertainty by pooling answers from several vision language models in one shared semantic space, with the central move being a split between total dispersion and disagreement among models. This post follows that split from the motivation to the measured numbers.

Why a pile of answers still leaves me uneasy

Hello, this is Patrick.

I have a habit of asking the same question to several models and comparing what comes back. I hope that when one fails, another will catch it. Yet once three answers pile up, a different worry starts. Are two right and one wrong, or are all three wrong in different ways. So does adding more voices by itself create trust.

That was the part that pulled me toward this paper. I had heard many times that combining heterogeneous vision language models can improve multimodal reasoning. But the trust question after the combination was still answered by looking at each model's own confidence, or by admiring how plausible the combined answer sounded. A confident single model does not make the pooled answer correct, and a plausible pooled answer does not mean the process was solid. CUSP tries to fill that gap with a number. It measures the uncertainty of the group through semantic opinion pooling, and it does so without fresh training or extra calibration data, which caught my eye first.

A meeting room makes the intuition fast. If four people reach the same conclusion from different evidence, the conclusion feels sturdy. If four people disagree and one option wins by vote, the visible answer is single but the unease of the whole room remains. CUSP tries to measure that whole room unease. It asks not who spoke loudest but how widely opinions spread in meaning, and whether the spread comes from each person's hesitation or from clashes between people.

Why meaning has to be pooled instead of wording

So what does pooling opinions mean exactly. Many readers will picture voting here. If three models say apple and two say pear, pick apple. On the surface that looks simple. In practice it is not.

The same apple answer can rest on different evidence and different phrasing. One model says apple from a round red body, another says apple from the leaf shape and the stem. The letters match but the scenes they saw are slightly different. The reverse also happens. A red apple and a ripe apple are different strings with the same conclusion. CUSP therefore refuses to count raw strings. It places responses on a shared semantic response space, a kind of map where similar meanings sit close and different meanings sit far apart. Each model's opinion goes on that map, and the opinions are combined into a pooled semantic opinion. The system answer and its uncertainty both come from that pooled opinion.

What interests me more is this shift from token probabilities to distributions over sampled meanings. Instead of opening each model and reading token scores, the method groups sampled answers by sense and reads the spread. People judge meetings the same way. We rarely score the transcript letter by letter. We group who meant what, then read the room. This is where I would be careful not to over-read the setup, because who decides closeness in meaning changes the shape of the map. Still the direction is clear. Look at the spread of senses, not the noise of wording.

From a systems perspective, the next question is obvious. If grouping by meaning costs a lot on every query, nobody will run it in production. That is why the paper stresses that no token logits and no calibration labels are needed. The work happens outside the models, from sampled answers alone. That choice connects directly to the commercial model experiments discussed later.

What changes when one spread becomes two numbers

Now I come to the distinction I found most engaging. CUSP reports two complementary signals. One is collective uncertainty, the dispersion of the pooled opinion. The other is Jensen Shannon divergence, or JSD, the conflict among model level opinions.

The problem comes next. If the two numbers drift apart, a practitioner will wonder which one to believe. The paper therefore shows an exact split for the unnormalized collective entropy. It equals the mean of the models' individual semantic entropies plus the JSD across models. In plain words, total spread equals average individual hesitation plus mutual disagreement. I think the more important result is the structure behind the numbers rather than any single value, because the cause changes the response. Whether everyone is confused, or everyone is confident but pointing in different directions, calls for different action.

Picture a late night incident channel with three engineers. If all three are unsure and hesitant, the lead gathers more evidence. If all three are confident but contradict each other, the lead checks whose premise is wrong before gathering anything. Both rooms feel tense, but the prescription differs completely. The CUSP split aims at the same judgment. Read the pooled spread to decide whether to stop, and read the JSD to see where the split happened.

That raises an enjoyable question. So which signal is more useful. The paper answers that it depends on the setting. Collective uncertainty led in the small model static regime, while JSD led in the evaluated commercial regime. It reads less like a search for one universal dial and more like advice to pick the right gauge for the situation.

Why a design that never opens the model feels welcome

When my team looks at a new AI architecture, we do not look at benchmark numbers alone. We ask what actually changed and what cost structure that change creates in a system. Through that lens, the training free character of CUSP and its freedom from token logits are not decorative phrases.

Many uncertainty methods want to look inside the model. They ask for per token distributions, temperature tuning on held out labels, or a head tailored to one structure. That is tidy in a lab, but it becomes a barrier when mixing closed models from several vendors. In front of a commercial model that never reveals logits, such methods cannot even start. CUSP walks around that wall. It samples answers from each model, places them in the semantic space, and pools them with outside computation. Any model that returns an answer, open or closed, can sit on the same map.

So how would you actually serve this. Sampling several answers per model adds cost, because a task that finished in one call now needs several calls per model, and latency and billing grow together. I therefore read this design as closer to an emergency lamp that turns on before painful questions than to a meter that runs on every query. The paper also leans that way, since it frames the use around prediction error detection and abstention ordering. Pass the questions you can answer, and route the risky ones to people or extra checks. In that flow the sampling cost has room to pay off.

There is another side to having no calibration labels. On one hand adoption is light, because a new domain needs no fresh labeled set for refitting. On the other hand the scale can wobble as domains shift, since a value like 0.7 may not mean the same risk in medical images and in street sign reading. That is why the paper speaks in ordering metrics like AUROC and AUARC. It tests the ability to rank risky items before safe ones, rather than the exactness of an absolute value.

What the numbers said when small models gathered

Let us turn to the experiments, starting with the static ensemble of small models. There collective uncertainty reached 0.764 AUROC for prediction error detection and 0.889 AUARC for abstention ordering. That beat simple baselines like majority voting and naive selection by 4.7 to 15.8 points, according to the paper. Unpacked, that means wrong answers tended to receive higher uncertainty in a fairly reliable order, and handing questions to people in that uncertain order lifted the accuracy of the remaining answers more steeply.

Anyone who has run small models knows that the same prompt can swing widely between runs. Small models are especially moody. Viewed alone, each answer wobbles with the day. Viewed together and pooled by meaning, the wobble finds a center. The pooled prediction itself also improved over the average single model by 5.6 to 13.0 points. Gathering improved the answer and revealed when the answer was risky at the same time. That pairing matters in operations, because accuracy and trust measurement drifting apart makes pipelines complex.

Should we accept the gain at face value. I paused here for a moment. In a small model group the skill gaps are wide, and those gaps may themselves supply the diversity that makes pooling look good. Wide diversity sharpens both the pooled answer and the spread signal. So the strong numbers in this range may reflect the design and the favorable conditions together. When conditions change, the ranking of signals can change too, and indeed the lead changes hands in the next setting.

Why the disagreement lens looked sharper among commercial models

In the evaluated commercial regime the lead belonged not to collective uncertainty but to JSD, the disagreement measure. It recorded 0.819 AUROC for prediction error detection and 0.910 AUARC for abstention, and its ranking of model conflict on hard questions reached up to 0.982 AUROC, as reported.

That contrast matters more than it first appears. Among small models the full spread including individual hesitation worked best, while among commercial models the clash between smooth but diverging voices mattered more. I connect this to the fluent and confident style of commercial models. Each model sounds polished on its own, yet slips in a different direction on hard questions. Then the average individual hesitation stays quiet while the mutual gap rings loud. JSD is the gauge that catches that ring.

Take a concrete scene. In a street photo task about a small shop sign, model A reads the restaurant name while model B reads the phone number of the next shop as the answer. Both speak with confidence. Judged by internal confidence alone, both might pass. Placed on the meaning map, the two opinions sit far apart and JSD jumps. An operator can send that question to human review or to an extra crop step. When the number maps so cleanly to action, an uncertainty metric finally speaks the language of the floor.

I still want to avoid stretching this point. Saying JSD was strongest in the commercial slice means strongest within that evaluated bundle and that question mix. When questions turn easy, models gather on the same answer and the disagreement gauge has little to do. When every question is hard and models fail differently, total spread and disagreement both swell and their separation power can cross again. I therefore read the result as practical guidance rather than a universal ranking. The more fluent the models, the less you should trust surface confidence and the more you should watch how they diverge.

The moment a pooled answer beats a lonely one

Let me return briefly to accuracy. The pooled prediction beating the average single model by 5.6 to 13.0 points can sound obvious, since crowds are supposed to help. Yet in the field, adding models does not always guarantee a better answer. If models share the same blind spot, pooling only hardens the wrong answer.

The pooling in CUSP feels different because it works at the level of sense rather than string voting. Different phrasings with the same sense add force in the same direction, which cancels small wording noise. When senses diverge, the method does not force a winner but leaves the split visible as a number. Answer selection and uncertainty measurement happen on the same map. I see that structure a bit like a shared error notebook. It keeps not only the chosen answer but also the trace of where the group hesitated.

That leaves a practical question. So how many models are enough. The evidence at hand does not support a firm rule on the count, so I will not state one. The direction is still clear. Too few makes the shape of the spread unstable, and too many raises cost. With commercial models in the mix, every call converts quickly into money and delay, so a layered route looks natural. Screen first with light models, then send only the hard cases to the heavier pool. The buying question is not how large the accuracy gain looks but what price paid for it, and that price will decide adoption.

What changes on a long trajectory with many steps

So far the scene was static, with several models answering one question. The paper goes one step further and tests the signal over full trajectories of a multistep multi-agent system. Subagent collective uncertainty ranked system failures at 0.619 AUROC and gave the best abstention ordering in that slice at 0.699 AUARC, as reported.

Those numbers look lower than the earlier 0.8 range, which may disappoint. I read them as honest instead, because choosing one answer and foreseeing the collapse of a long run are tasks of different difficulty. A small misunderstanding early can amplify later, or it can get lucky and repaired midway. Ranking whole trajectories from partial spread alone is bound to score lower. That it still gave the best ordering in that slice suggests that spread in meaning carries the smell of failure even across long work.

Organizations of people behave similarly. A final report can look tidy while the interim notes show where the team struggled. If subagent opinions diverged widely at several steps, a smooth final answer may hide a forced patch somewhere. If opinions gathered steadily at each step, trust in the final answer can run a little deeper. The paper looks at the whole trajectory with the same concern. It tries to watch the process, not only the final answer.

The conditions still deserve care. An ordering of 0.619 beats chance, but it is too thin to drive automatic halts or automatic deployment alone. I would therefore place this signal in triage rather than as a lone breaker. Let people open the most uncertain trajectories first. Reviewing in risky order instead of reviewing everything already makes the night shift far less busy.

What to keep and what to handle with care

Let me widen the view to close. What remains after this work. I would keep three things. First, it measured the unease of the group directly, beyond single model confidence and beyond the plausibility of the pooled answer. Second, it made that unease readable as individual hesitation plus mutual disagreement. Third, it did the job from outside answers alone, without token internals or calibration data. Together they widen the practical choice, because open and closed models can be mixed while risky questions are ranked on the same scale.

That scale should not be trusted as an absolute value. Good ordering means skill at placing risky items first, not a promise that 0.62 means the same danger in every domain. The quality of the semantic space also moves the scale, since a different yardstick for grouping senses changes the size of the spread. I see that less as a weakness and more as the next place to dig. Which yardstick reveals failures best, and what must stay fixed so the scale survives a domain shift, are exactly the follow-up questions.

What I would watch next is twofold. One is operational evidence on how this signal pairs with human review, meaning at which threshold how many cases were routed and how many painful failures were stopped, rather than one more AUROC line. The other is a finer cut of long trajectories, tracing which step diverged before the final failure, because that would support midcourse intervention as well as abstention. After learning to measure the unease of the whole meeting, the next move is to point at the passage where it creaked.

In a multistep system the confidence of one agent does not end with that agent. It becomes the premise of the next step, and that premise becomes the frame of the final answer. Reading the total unease as individual hesitation plus mutual disagreement is therefore close to an attitude toward work, not only a scoring trick. Teams that know where they wobbled tend to travel further, and in that respect meetings of models do not look so different from meetings of people.