Fixing Structured Outputs in a 350M Model With 100 GRPO Steps

The piece covered here is Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps, a blog post published by Hugging Face. The original is linked under the title.

The claim is that tuning a 350M model for only 100 GRPO steps with group normalized verifiable rewards can lift valid structured output rates above 90 percent on in-distribution tasks.

Structured output is no longer optional. Citation blocks in retrieval augmented generation, tool calls in agent loops, records in extraction pipelines, and form autofill in user interfaces all require machine readable form. An answer that reads smoothly for a person and an answer that parses cleanly for code follow different rules. One missing quote, one stray comma, or one unclosed brace throws an exception downstream. Good content with broken form still fails. So structured reliability is not a bonus feature. It is a precondition for deployment. That is why the focus on 350M parameters matters. If a model that is small, fast, and cheap to run can hold form, the economics of many production pipelines change.

Start with the reality of small models. A 350M class model has clear advantages in latency and cost. It runs comfortably on a single GPU, scales more easily under concurrent load than a large model, and fits edge devices, on premise servers, and cost sensitive batch extraction. In places with data residency rules or tight budgets, a small self hosted model may be the only option. But the baseline small model is inconsistent on JSON validity. It handles flat simple schemas well enough, then breaks when fields grow, nesting deepens, or enum constraints appear. Unclosed braces, unquoted keys, invented fields outside the schema, and numbers written as strings keep recurring. Prompting helps to a point, but prompting does not change habits. Long format instructions on every request inflate input tokens, and form still loosens near the end of long outputs. The starting point of this post is therefore that the habit itself needs training.

Why structured output tests small models

Structured output asks for meaning and discipline at once. The model must understand the request and pour that understanding into a fixed container. Each demand creates a different failure. Misunderstood meaning gives wrong values. Broken discipline gives parse failure. Parse failure is harder to repair. A slightly wrong value can survive with postprocessing or partial retry, but broken JSON must be discarded wholesale. Retries raise latency and cost. That is why a few percentage points of validity matter so much in operations. Eighty percent validity means throwing away one in five responses. Ninety five percent validity means throwing away only one in twenty. Retry budgets, timeout design, and user experience all rest on that number.

Small models suffer most where length meets detailed constraints. The longer the output, the more format error probability accumulates. The opening looks fine, then a comma goes missing near the end or an array never closes. Schema constraints are also hard to convey through natural language prompts alone. The split between required and optional fields, string patterns, numeric ranges, enum values, and nesting depth grows long and vague when written as prose. The model must interpret that vague description on every call. A model that absorbed grammar during training behaves more stably than a model that reasons from instructions each time. The GRPO approach here moves from the second regime to the first. Instead of repeating explanations, it rewards form holding behavior until it becomes habit.

Evaluation has layers too. JSON parseability is the lowest bar. If the parser can read it, stage one passes. Above that sits schema validity. Types, required fields, and enum values must match. Above that sits field level correctness. Values must fit the request and the facts, with dates, units, and identifiers in shape. If a lower layer breaks, upper layers cannot even be judged. That layered shape makes structured output ideal for staged rewards. Separating parse failure, schema violation, and value error tells the model where it went wrong. That layering becomes the skeleton of the reward design discussed later.

How GRPO learns inside the group

GRPO, or group relative policy optimization, samples several completions for one prompt and learns from their relative standing. For each prompt it draws 4 to 8 completions, assigns each a verifiable reward, normalizes those rewards within the group, and updates the policy toward the relatively better ones while moving away from the relatively worse ones. The reference is internal. Brothers from the same prompt set the curve. This structure removes the need for a separate value critic. There is no value function to train, which cuts memory, compute, and tuning burden, and it works cleanly where rewards come from deterministic checks.

The intuition behind group normalization deserves care. Prompts differ in difficulty. Some requests have simple schemas where most completions pass. Others have deep nesting where most fail. With absolute scores alone, a small gap on an easy problem and a large gap on a hard problem get hard to weigh together. Normalizing inside the group gives each prompt its own grading curve. On easy problems only near perfect completions earn advantage. On hard problems the least broken completion earns advantage. The model learns from relative quality under identical conditions rather than absolute difficulty. Updates stay steadier when difficulty varies across batches. That steadiness matters greatly when a small model must rise within only 100 steps.

Living without a critic has practical meaning. A critic is another thing to train, with its own memory, compute, and instability. Placing one beside a 350M policy strains a single GPU budget. GRPO spends more on sampling instead and saves on the critic. Drawing several completions costs compute, but reward computation in structured output is deterministic code, so the whole pipeline stays simple. Parse, validate against the schema, compare fields, assign the reward, and let within group ranking supply the learning signal. Fewer dials help in short training. With fewer moving parts, the run is less likely to lose direction inside a 100 step budget.

Group sampling is not free. Drawing several completions per prompt multiplies generation per step. Batch size times group size sets the generation load. But the math here converges on total time and total resources, not per step cost. If the run ends in 100 steps and fits on one GPU in under an hour, some extra generation per step is affordable. It is nothing like training a large model for days. In short, GRPO offers a sensible trade for this task. Drop the critic, add samples, and use the relative order of verifiable rewards to steady a small model fast.

Why 350M, and what smallness buys

The number 350M is deliberate. It sits where latency, cost, and deployment flexibility meet. Consider per call price and wait for large models. In high frequency structured extraction with fixed form, per call cost dominates the budget. If a small model holds form reliably, the same work can cost a fraction per call. In on premise networks, edge hardware, or settings where external large model calls are blocked, carrying a small model inside is often the only path. The gap is that baseline validity is low, and this post claims that 100 steps can close it.

A low starting point can be read as headroom rather than defect. Large models already absorbed much form discipline through vast pretraining and instruction tuning, so extra training adds less. Small models show clear deficits in form discipline, which means well designed rewards have more to fix. Repeated patterns like unclosed structures, quote mistakes, and appended explanations are exactly the regular errors that reinforcement learning catches well. If the model already grasps much of the meaning, the remaining work is output habit, and habit responds directly to verifiable rewards. The reported jump past 90 percent suggests that small models hold surprising potential precisely on form tasks.

Deployment gains from 350M can be spelled out. Low memory means larger batches or more replicas on the same card. Higher throughput eases peak hour handling. Short latency keeps agent loops responsive when structured calls repeat inside one task, because one slow tool call slows the whole agent. Fast structured calls widen agent design freedom. Small models are also light to version and roll back. When schemas evolve, retraining stays cheap, so the model can evolve with the schema. Combined with a one GPU one hour loop, that lightness points to routine refreshes whenever schemas or tools change.

Reward as specification

The core of the method is the reward function. Rewards tell the model what counted as good, so they act as a specification. Here the reward stacks three deterministic checks. First comes JSON parseability, whether a parser can read the output. Second comes JSON Schema validity, whether types, required fields, enum values, and patterns match. Third comes field level correctness, whether values fit the request in content, not just shape. The three layers mirror the evaluation ladder from the earlier section. Broken parsing earns low reward, clean parsing with schema mismatch earns middle reward, and full schema plus correct values earns high reward. The model climbs that slope toward disciplined form.

Deterministic rewards matter. Scores come from reproducible code, not human taste or a large judge model. The same output always earns the same reward. Low noise keeps signal clear across a short 100 step run. Rewards also connect directly to the task contract. When the schema changes, validation code changes and rewards change with it. Training and deployment then share one ruler. That closes a common gap where training scores rise on a proxy while production still fails parsing. Saying the reward is the specification is therefore literal. Validation code doubles as reward code.

A subtle point is the gradient of partial credit. Pure binary rewards of all or nothing leave an early model with constant zeros and nothing to learn. Staged rewards fix that. Small credit for parsing alone, more credit as schema items match, larger credit when values match, so even failure carries direction. Learning to close braces alone lifts reward a little, and the model holds that behavior while reaching further. Habits like trailing chatter, excess code fences, and apology sentences earn quick penalties through parse failure and fade fast. The shape of the reward curve sets learning speed.

Balance between strict and lenient also matters. Overly strict schema checks can push the model toward empty shells. Dropping fields still parses but loses required field credit. Inventing fields violates schema. Between those pressures the model moves toward filling needed fields completely. Value checks block placeholders and empty strings from gaming the system. Each layer blocks a different shortcut, and together the three layers push toward honest completions. That is why the design uses all three. Each layer guards one gaming path, and the stack guides the model to real improvement.

The 100 step design and what one hour on one GPU means

One hundred steps declares brevity. Reinforcement fine tuning often runs thousands of steps, yet this post says 100 is enough. Three reasons make that plausible. First, the base model already understands much meaning, so the run fixes habits rather than teaching from scratch, which shortens the distance to travel. Second, rewards are deterministic and dense, with clear signals from parsing, schema, and values on every step. Third, group normalization lowers variance, so direction stays stable. When those three overlap, short training reaches deployment quality. The point is not that longer runs never help, but that a short run already lands in a usable zone.

Finishing on a single GPU in under an hour changes access. Teams without large clusters can try, reproduce, and adapt the method. University labs, small product teams, and internal infrastructure groups all qualify. Access widens impact. A method that needs dozens of cards stays a picture on the wall for most practitioners. A one card one hour loop invites repetition. Refresh when schemas change, refresh when domains shift, refresh when failures accumulate. Fine tuning turns from a project into a routine.

In operational language, sampling 4 to 8 completions per prompt is the central batch dial. Larger groups stabilize advantage estimates but cost more generation. Smaller groups cost less but wobble more. The reported range marks a practical middle where a handful of samples already works, without needing dozens. Short step counts also lighten hyperparameter worry. Long runs invite late collapse, while 100 steps eats the early rapid gains and stops before overfitting deepens. Brevity doubles as regularization.

How to read the jump past 90 percent

The headline number needs careful reading. Valid structured output above 90 percent on in-distribution tasks follows an inconsistent baseline. Before tuning, the small model swings with conditions, doing acceptably on some schemas and breaking often on others. After tuning, spread narrows and performance settles high. Mean rises and variance shrinks together. Operators often welcome the second change more, because worst case failure rate drives incident tickets. Read the 90 percent as both average gain and worst case gain.

The phrase in-distribution carries weight. Strength inside trained schema families does not guarantee equal strength on unseen shapes. Production schemas keep moving. Fields get added, nesting deepens, enum sets grow. The value here is solid inside familiar families and needs verification outside them. Treat the public number as evidence of direction and your own schema numbers as evidence of adoption. Confusing the two leads rollout astray.

Separate validity from usefulness as well. Parsing plus schema pass means a machine can read the output, which is not identical to values being right. Even with value checks in the reward, correctness needs task specific measurement. One wrong date, one wrong unit, or one wrong identifier can pass shape checks while failing the job. Adoption tests should therefore track three rates together, parse rate, schema pass rate, and field correctness rate. All three must rise for the gain to be real. If only parsing rises while values stall, form improved but content did not. The three layer reward exists for that reason, and measurement should mirror it.

What the KL penalty protects

A common reinforcement tuning failure is collapse. Chasing reward alone can erase general language ability. In structured tasks collapse has a clear face. The model becomes a brace parrot that opens JSON for any request, lists keys when asked for explanation, and wraps greetings in schema. Rewards rose while the model broke. A KL penalty against the reference policy acts as the leash that stops that slide. If the tuned policy drifts too far from the starting model, it pays cost, so form changes while fundamentals stay.

Think of KL as an anchor if the math feels distant. The ship leaves harbor but moves only within rope length. Too short a rope blocks progress. Too long a rope lets the ship drift. The KL coefficient sets that length. In a 100 step run the anchor matters more, because early steep reward climbs can yank the model hard in one direction. JSON habits can over strengthen before the run ends unless the anchor holds. The result preserves a dual ability. Structured calls hold form, normal chat still talks normally. Small models gain specialty without losing generality.

In practice, check KL effects with regression tests. Do not watch structured metrics alone. Measure instruction following and simple question answering alongside, and confirm that scores barely move from before tuning. If general ability dropped, raise the KL coefficient or stop earlier. If structured gains stall, loosen KL slightly to allow movement. Translated to operations, reward is the accelerator and KL is the steering. Both hands are needed to arrive.

Remaining risks and limits

Strong results cast shadows. Three stand out here, reward hacking, overfitting to training schemas, and degradation on unseen nested structures. Reward hacking means exploiting blind spots in checks to raise scores without real quality. Empty values, safe boilerplate, or schema passing filler can look good to code while saying little. Even with value checks, whatever validation ignores becomes the new direction of learning. Since validation is the reward, gaps in validation become gaps in training. Hardening validation code matters as much as training itself.

Overfitting to training schemas is the second shadow. Past 90 percent inside familiar families may not transfer to new families. The model may have memorized training habits rather than general structure. New field names, new depths, and new enum patterns can break memorized routines. Production schemas live and move as features and tools change, so an overfit model demands retraining each time. If retraining is a one hour routine, that demand is manageable, but measurement must include held out schema sets that never appear in training. Tracking in-distribution and out of distribution rates side by side guards against overconfidence.

The third limit is deep nesting. Flat objects are easy, while arrays inside objects inside arrays stay hard. Closing management stretches, and field ownership across depths confuses a small model. One hundred steps can fix habits but cannot raise the ceiling of expressive power indefinitely. Strange deep structures will still fail. That calls for design answers alongside training, such as length limits, split generation, or staged assembly. Generate in parts and join them rather than forcing one giant nested output. Training plus task decomposition travels further than training alone.

A checklist for the field

How should teams carry this into production. A five step checklist helps. First, freeze the schema and write validation first. Build parser, schema checks, and value comparison before training, because that code is both reward and evaluation. Shaky validation means shaky training. Second, collect failures. Gather current parse failures and schema violations and study their spread, then set reward weights accordingly. If closing errors dominate, weight parsing. If invented fields dominate, weight schema. Third, start group size between 4 and 8. Begin small and grow on stability. Fourth, attach KL and track regressions. Watch structured metrics and general instruction behavior together. Fifth, hold out schemas. Keep test schemas out of training and record pass rates there every run. Proceed to rollout only when all five boxes read well.

Prompt habits should shift too. If long format instructions were attached to every request before training, shorten them after training toward schema identifiers alone. Learned habits reduce the need for repeated explanation, and shorter inputs cut cost and latency together. Keep postprocessing guardrails on outputs, such as no chatter and no stray fences. Safety nets stay even after habits improve. Set a retry policy for parse failure, such as one retry at lower temperature followed by full regeneration rather than partial repair. At 90 percent plus validity, a single retry cuts perceived failure sharply.

At team level, start with a small pilot. Pick the highest frequency pipeline with the most stable schema and compare parse rate, schema pass rate, and field correctness on the same input set before and after tuning. Record cost and latency shifts alongside. Expand to the next pipeline only when numbers hold. Leave fast changing schemas for later. Lock gains on stable ground first, then move to shifting ground. Since this method favors light repetition, validation code and training configs from the pilot become reusable assets for the next schema.

Learning to trust small models with structure

The direction is now clear. The issue was habit, not size. Given precise feedback on what counted as good, a 350M model learns form discipline in a short time. Relative ranking inside groups, deterministic validation as reward, and an anchor to the reference policy make that learning possible. The numbers around 100 steps and one GPU hour are therefore not mere efficiency claims. They declare access for smaller teams and repeatability whenever schemas change.

Limits remain concrete. Validation gaps need continuous patching, unseen schemas need continuous testing, and deep nesting ceilings need design help. But those tasks are specific rather than vague. Harden validation code, add held out evaluation, and split hard structures into parts. Visible next work signals a maturing method. As structured output shifts from specialty trick to default pipeline, this post adds one more reason to trust small fast models. The next time a schema changes, consider a 100 step tune before calling a larger model. That short run may be the cheapest way to move production failure rates.