RL Alignment: GRPO & RLHF
Choose an alignment method from observable behavior, reward failure modes, and rollback boundaries—not fashion.
Design task suitability, rollouts, rewards, GRPO/RLHF choice, reward hacking controls, evaluation, and recovery together.
01
Problem definition
Applying weak rewards to tasks without reliable verification teaches the model scoring loopholes rather than intent.
Higher rollout throughput is meaningless when policy versions and sample quality are mixed.
- Outcome or process quality can be judged with a repeatable reward
- Post-SFT behavior differences can be checked on a clear held-out evaluation
- The reward does not represent the human decision
- Rollout cost and failure recovery cannot be bounded
RL Alignment: GRPO & RLHF: system plate
- 01
State 1
Classify tasks as outcome-, process-, or preference-verifiable to constrain method choice.
- 02
State 2
Fix policy version and initial state within rollout groups to make samples comparable.
- 03
State 3
Decompose reward signals and track length, formatting, and shortcut optimization as distinct failures.
- 04
State 4
Join held-out behavior evaluation with checkpoint promotion and rollback in one release gate.
FEEDBACKFailed acceptance returns evidence to the first controlled stage: Valid rollout rate.
- The workflow begins with Classify tasks as outcome-, process-, or preference-verifiable to constrain method choice..
- It reaches an acceptance decision through Valid rollout rate.
03
Design method
Choose an alignment method from observable behavior, reward failure modes, and rollback boundaries—not fashion.
- 01
Stage 1
Classify tasks as outcome-, process-, or preference-verifiable to constrain method choice.
- 02
Stage 2
Fix policy version and initial state within rollout groups to make samples comparable.
- 03
Stage 3
Decompose reward signals and track length, formatting, and shortcut optimization as distinct failures.
- 04
Stage 4
Join held-out behavior evaluation with checkpoint promotion and rollback in one release gate.
04
Application scenarios
Outcome or process quality can be judged with a repeatable reward
Applying weak rewards to tasks without reliable verification teaches the model scoring loopholes rather than intent.
- APPROACH
- Classify tasks as outcome-, process-, or preference-verifiable to constrain method choice.
- BOUNDARY
- A higher reward is not interpreted as completed alignment or safety.
Post-SFT behavior differences can be checked on a clear held-out evaluation
Higher rollout throughput is meaningless when policy versions and sample quality are mixed.
- APPROACH
- Fix policy version and initial state within rollout groups to make samples comparable.
- BOUNDARY
- Samples without comparable policy versions are excluded from training and evaluation.
05
Design choices
| Decision | Gain | Cost | Watch |
|---|---|---|---|
| Outcome or process quality can be judged with a repeatable reward | Classify tasks as outcome-, process-, or preference-verifiable to constrain method choice. | A higher reward is not interpreted as completed alignment or safety. | Valid rollout rate |
| Post-SFT behavior differences can be checked on a clear held-out evaluation | Fix policy version and initial state within rollout groups to make samples comparable. | Samples without comparable policy versions are excluded from training and evaluation. | Reward-to-human agreement |
RL Alignment: GRPO & RLHF: system plate
- The reward does not represent the human decision
A higher reward is not interpreted as completed alignment or safety.
- SIGNAL
- Valid rollout rate
- MITIGATION
- Fix policy version and initial state within rollout groups to make samples comparable.
- Rollout cost and failure recovery cannot be bounded
Samples without comparable policy versions are excluded from training and evaluation.
- SIGNAL
- Reward-to-human agreement
- MITIGATION
- Decompose reward signals and track length, formatting, and shortcut optimization as distinct failures.
- The workflow begins with Classify tasks as outcome-, process-, or preference-verifiable to constrain method choice..
- It reaches an acceptance decision through Valid rollout rate.
07
Validation plan
| Measure | Method | Pass condition | Caveat |
|---|---|---|---|
| Valid rollout rate | Classify tasks as outcome-, process-, or preference-verifiable to constrain method choice. | Repeated runs satisfy the acceptance threshold agreed during discovery | A higher reward is not interpreted as completed alignment or safety. |
| Reward-to-human agreement | Fix policy version and initial state within rollout groups to make samples comparable. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
| Held-out behavior regression | Decompose reward signals and track length, formatting, and shortcut optimization as distinct failures. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
08
Constraints and failure conditions
The reward does not represent the human decision
A higher reward is not interpreted as completed alignment or safety.
Rollout cost and failure recovery cannot be bounded
Samples without comparable policy versions are excluded from training and evaluation.
09
Engagement model
- 01
Diagnosis
PattyAnalyze the current system and its failure signals.
ClientProvide representative work, data boundaries, and operating constraints.
Valid rollout rate - 02
Design
PattyFix policy version and initial state within rollout groups to make samples comparable.
ClientConfirm owners and acceptance criteria.
Reward-to-human agreement - 03
Validation
PattyDecompose reward signals and track length, formatting, and shortcut optimization as distinct failures.
ClientMake the production-transition or stop decision.
Held-out behavior regression
10
Durable deliverables
- RL Alignment: GRPO & RLHF decision record
- Choose an alignment method from observable behavior, reward failure modes, and rollback boundaries—not fashion.Client-owned · Patty-reviewed
- Validation harness and acceptance criteria
- Valid rollout rate · Reward-to-human agreement · Held-out behavior regressionJointly maintained
- Operations and recovery runbook
- A higher reward is not interpreted as completed alignment or safety. · Samples without comparable policy versions are excluded from training and evaluation.Operating-team owned
11
Terminology
- RL Alignment: GRPO & RLHF
- Design task suitability, rollouts, rewards, GRPO/RLHF choice, reward hacking controls, evaluation, and recovery together.
- Acceptance criterion
- Valid rollout rate
- Operating boundary
- A higher reward is not interpreted as completed alignment or safety.
REFERENCES
References and primary material
- TRL GRPO Trainer
Primary material for the method and terminology.
- DeepSeekMath
Primary material for the method and terminology.