Should SFT Learn Every Token, What Trimming Teaches Us About Math Reasoning

The paper I am covering today is Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning (2609.09707), published on arXiv. It diagnoses uniform cross entropy loss in SFT as a poor fit for mathematical reasoning and proposes TrimSFT, which scales the loss by the logit gap between the gold token and its strongest competitor. The original is linked under the title.

What happens when we make the model rehearse what it already knows

What interests me more than the method itself is the discomfort behind the title. When I fine tune on math solutions, I keep seeing the same odd scene. The model already predicts particles, formatting symbols, and familiar calculation patterns with high confidence, yet those tokens still receive the full loss at every step. So I started asking whether that is really learning. If you have trained on long proof traces, you have probably seen the same thing. The loss looks healthy because it is dominated by obvious tokens, while the branching points where the model actually hesitates get buried in the average.

So where is the problem. The diagnosis in the paper is simple. Standard SFT applies uniform cross entropy loss to all target tokens, even though tokens in mathematical reasoning carry very unequal learning signals. Some tokens are already mastered, some are still far from decided, and some sit in a boundary zone where teaching lands well right now. I like this framing because it restates training as a problem of attention allocation rather than data volume.

People study in a similar way. When you prepare for an exam, you do not rewrite the words you already memorized ten more times. You spend time on the two or three patterns that keep confusing you, and that is what moves the score. Yet our SFT has been closer to the opposite. It sharpens what is already known and piles pressure on what is still shaky. This paper questions that habit directly. And the fact that the doubt shows up in the numbers is what carries the rest of this post.

Why SFT wants to teach every token the same way

The default behavior of SFT looks reasonable from the outside. Given a gold sequence, it pushes up the probability of the gold token at each position, and it treats every position as equally precious. But in long reasoning traces such as math solutions, that fairness turns costly. A solution contains many pieces the base model already knows, including formatting tokens, repeated connective words, and hardened calculation patterns. When we teach those pieces with the same strength, the model learns to be overconfident about old knowledge rather than learning new reasoning. That is what the authors call over sharpening.

The opposite happens on low confidence tokens. When the model sits in a very different distribution from the gold answer and we still apply the full loss there, the gradients can swing and training turns unstable. So uniform SFT loses on both sides. On one side it polishes what is already learned too aggressively, and on the other side it applies excessive pressure where the model is not ready. I nodded when I read this diagnosis. I have often seen a loss curve that looks like healthy training while reasoning ability stays flat once you probe it closely.

That raises an interesting question. Which tokens should get more effort and which ones should be allowed to rest. In human studying the answer is clear. You set aside what is too easy and what is too hard for now, and you spend time in the middle stretch that feels almost within reach. The paper moves that intuition down to the token level. And it pulls out the logit gap as the measuring stick.

What the distance between first and second place tells us

What I found most engaging here is that the learning signal is read as a gap rather than a probability. At each position the authors look at the gold token logit minus the strongest competitor logit. When that logit gap is large, the model already places the gold answer clearly ahead of the runner up. When the value is small or negative, the gold answer is still tied or trailing. Instead of collapsing everything into one probability, this view keeps track of who came second.

That gap carries more information than it first appears. A very large gap is close to a signal that the token is already learned. More teaching there is likely to add overconfidence rather than understanding. A very small or negative gap signals that the model is not yet ready to make that decision, so a large gradient now may point in the wrong direction. The middle region is different. The gold answer has started to rise but has not hardened yet, so teaching in that window sinks in best. I think the structure behind the number matters more than the number itself. Difficulty is defined here as a relation between the token and the current state of the model, not as a fixed property of the token.

This is where I would be careful not to over read the experiment. A small gap does not always mean meaningfully hard. At points where several word choices are legitimately open, the runner up will naturally sit close behind. Even with that caveat, the gap works like a useful thermometer. It asks for each token whether now is the right time to teach, while the older approach kept the same temperature across the whole sequence.

Could trimming both ends be the answer

TrimSFT does what its name suggests. It trims supervision from both extremes and concentrates learning in an intermediate logit gap region. Tokens that are already mastered get less force so they do not turn overconfident, tokens that are still very shaky get less unstable pressure, and the training budget of gradients flows into the middle zone that is ready to learn. Think of it as pruning a garden. You leave the mature branches alone, you avoid touching the fragile shoots, and you tend the middle branches that are actively growing.

A natural objection comes to mind right away. If we trim hard tokens, is the model just avoiding hard reasoning. I paused at this point too on my first read. But the picture in the paper is softer than dropping tokens. It is not a hard filter that sets weights to zero. It places a smooth weighting curve centered in the middle, so hard tokens still keep a small weight and easy tokens still keep a small weight. It lowers the volume rather than switching off the sound. That distinction matters more than it sounds. It hands the leadership of the gradients to the middle zone without badly distorting the training distribution.

One more point worth naming is that the method does not change the structure of the reasoning trace. Instead of selecting data or reordering it, it shines the light differently on each token within the same data. That makes it easy to place on top of an existing SFT pipeline. There is no need to collect data again or relabel anything, and that low friction matters a lot in practice.

A light prescription built from one Gaussian

Let me walk through the mechanism in enough detail that you could picture it running. TrimSFT multiplies the SFT loss by a Gaussian weight centered at margin m. When a token logit gap sits near m, the weight is close to one. As the gap moves away from m, the weight falls off smoothly. The bandwidth tau controls how steep that falloff is. A small tau concentrates sharply on a narrow middle band, while a large tau spreads generously over a wider band. I see that simplicity as part of the appeal. The weight comes straight from logits that were already computed, with no reference model and no extra forward pass.

From a systems perspective, the next question is obvious. Do we need to host another model. The answer is no. Many recent weighted SFT variants ask for a reference model or an additional forward pass, but TrimSFT needs neither. It computes the weight from the training model logits alone, so memory and compute cost barely move. When my team looks at a new fine tuning method, we look beyond the benchmark number and ask what changes in the system and what cost structure that change creates. On that test this method is pleasantly light.

Light does not automatically mean effective, of course. The Gaussian shape itself is a strong assumption, because it forces symmetry onto a problem that may not be symmetric. That is why I kept asking why the curve should be bell shaped in the first place. The paper does not fully settle whether an asymmetric shape might fit better, yet I still find this a sensible starting point. This choice expresses the idea of focus in the middle and trim at both ends with only two knobs, m and tau. Since there are only two knobs, researchers and practitioners can sweep the design quickly, which is welcome in practice. But is this light prescription really the right one? That is what the bandwidth experiments still need to settle.

Why the structure caught my eye before the numbers

Let me step away from the numbers for a moment and look at the structure. TrimSFT does not only change the size of the loss. It moves the center of gravity of where gradients come from. Under uniform SFT, obvious tokens win by count and take over the total gradient. In a long solution, repeated tokens such as Let or therefore plus hardened calculation patterns each add a small gradient at every step and pull the whole direction. Meanwhile the tokens that mark real branching points, such as a rephrased equation or an auxiliary construction, are few in number so their voice stays small. TrimSFT flips that majority rule. It turns up the volume of the fewer tokens that are worth learning right now.

A classroom analogy makes this concrete. A class that makes the top student solve the same homework ten more times while throwing calculus at a student who has not learned fractions is not a good class. The right move is to give the strong student the next level and give the developing student a stepping stone. TrimSFT uses the logit gap to sense which classroom each token currently sits in, then spends teacher time on the middle classroom. (That middle classroom is not a fixed vocabulary set. It is a moving window that shifts as training proceeds. What is middle today can become easy tomorrow.)

I regard that structural shift as more than a trick. It moves SFT from copying gold answers toward state dependent teaching. The idea that the same gold sequence should be taught with different strength depending on model state is a small philosophical move from data centered training toward state centered training.

What happened across six models and five benchmarks

Now to the evidence. The authors evaluated six base models across the Llama, Qwen, and DeepMath families on five math reasoning benchmarks. I read that setup as an attempt to show generality across families rather than luck on one or two models. The outcome was fairly clear. TrimSFT took the best average performance on five out of six models, with gains of up to 26.9 points over SFT on MATH500. Readers who have watched numbers swing wildly between small config changes will know why I did not skim past that margin. A gap of 26.9 points is too large to dismiss as noise.

Still, that number should be read with its conditions in view. Large gains on MATH500 tend to open up more when the starting point of the base model is lower. In other words, TrimSFT helps most where plain SFT was struggling badly. That is both a strength and a limit. It is excellent at lifting the floor, but the size of the win may shrink for models that already sit at a high level. The claim of first place on five out of six models should be read in the same spirit. It means one model missed the top spot, and asking which family it belonged to and where it slipped would reveal the boundary of the method.

So where should this be tested next. What I would watch next is whether the same pattern repeats beyond math, in tasks that also need long reasoning such as code or logical writing. Math traces have unusual repetition and regularity, so the middle zone lens may fit them especially well. I will return to that point at the end.

Why trimming only one side leaves something on the table

The part I appreciated most was the half trim experiment. Instead of trimming both sides, the authors tried variants that removed only one side, and either choice left an inferior trade off compared with full TrimSFT. If you trim only easy tokens, overconfidence falls but the unstable pressure from hard tokens remains. If you trim only hard tokens, stability improves but overconfidence from easy tokens remains. Each side alone tells half the story, so both need to be handled for the full picture to hold. That is the authors reading, and I find it persuasive.

The observation about bandwidth tau belongs with this result. The width tau mattered more than the exact location of margin m. For practice, I think this is the most usable detail. It suggests spending time first on how wide the middle view should be rather than tuning m to extra decimal places. When tau is too small, too few tokens get taught and learning slows. When tau is too large, the method slides back toward uniform SFT and trimming loses its meaning. Finding a reasonable width decides much of the outcome.

That outcome says more than it seems. The token learning problem is not caused by one extreme alone. Both extremes jointly create the trouble. Fix only one side and the other side keeps eating the gradients. So a symmetric prescription that trims both sides earns its logic. Perfect symmetry may not always be the final answer, but starting with both sides in view looks right based on these ablations.

How to actually use this in a training line

So how would you actually serve this in a learning pipeline. The answer from TrimSFT is refreshingly plain. You add one line of per token weighting inside the existing SFT loop. There is no need to rebuild data, no extra GPU for a reference model, and no added forward pass. At each step you measure the gap from logits you already computed, multiply by the Gaussian weight, and run backward as usual. For teams that feel nervous about touching a fine tuning job that runs overnight, this scale of change is easy to try.

The operational difference is larger than it sounds. Unlike preference learning setups that keep two models in memory or require serving a separate reward model, this approach does not change the shape of the pipeline. Batch construction, checkpoint strategy, and evaluation routines can stay as they are while only the loss weighting changes. Rollback is also easy. Turn off the weighting and you are back to plain SFT. I rate this method above its lab score for that reason. The adoption cost is low enough that even a small team can test it right away.

There is still a caution for the field. Values of m and tau need to be found again for your model and data distribution. The numbers in the paper are not guaranteed to carry over to your data. In particular, when base models differ a lot in skill, the whole distribution of logit gaps shifts, so the same m can point at a very different band. A good first pass is to start with a generous tau, log the weight distribution, and check by eye whether the middle band actually holds branching tokens. Look at whether the distribution makes sense before celebrating a higher score.

The questions we have not asked yet feel more interesting

Time to return to the question I deferred. How far can this travel beyond math. Here I felt some doubt. Math solutions carry strong repetition and formulaic shape inside gold traces, so the middle zone lens may fit them unusually well. In code generation where the reward of execution arrives late, or in open ended writing with many valid answers, the meaning of the logit gap can shift. A narrow gap might mean the right time to learn, or it might mean a natural fork where several answers are fine. Telling those two apart is harder outside math.

Another open point is the late stage of training. As learning proceeds, yesterday middle becomes today easy and the whole gap distribution drifts to the right. Can a fixed m follow that drift to the end. Perhaps m will need scheduling or automatic movement tied to quantiles of the distribution. The paper shows that a fixed m already wins enough, but an adaptive margin looks more natural for long runs. That is something I want to see in follow up work.

That leaves a human question. As the job of measuring when to teach each token becomes more automatic, the job of deciding what to teach grows more important. Which traces count as gold answers, which solution style becomes the standard, which mistakes deserve patience and which ones deserve strict correction. TrimSFT answers how to teach, but what to teach remains ours to choose. I actually think that remaining share is the larger one.

Trimming as a way of teaching

A question that first sounded provocative now feels quite practical. Should SFT really learn every token. The answer in this paper leans toward no. Trim what is already mastered so it does not breed overconfidence, trim what is still shaky so it does not breed instability, and gather teaching in the middle stretch that is currently within reach. The result of best average score on five of six models and up to 26.9 points ahead on MATH500 gives that answer real weight. Add the finding that half trimming fell short and that bandwidth mattered more than exact margin position, and the story grows from a trick into a philosophy of training focus.

Closing this paper reminded me of an old study habit. Skim quickly past pages you already know, set aside for now the pages that make your hands shake, and stay longer on pages that feel almost known but not quite. TrimSFT moves that habit down to the token level. Open problems remain around generality beyond math and scheduling over long runs. But for SFT on mathematical reasoning, the paper makes a clear case that trimming can be a path to learning more. The next time I look at a training loss, I will ask who is speaking inside the total before I ask what the total says.