Pretraining & CPT
Continued training controls what a model learns without silently sacrificing what it already knows.
A CPT discipline spanning corpus diagnosis, mixing, tokenization, distributed runs, checkpoints, and serving acceptance.
01
Problem definition
CPT is not automatically justified when a general model struggles with organizational language. Retrieval, SFT, and prompting must be ruled in or out first.
Small corpus biases can affect long-term knowledge, instruction following, and safety together. Discovering the wrong training direction late is costlier than a single run.
- A persistent domain-language and knowledge gap is not resolved by retrieval alone
- A long-term model strategy includes clear ownership and deployment boundaries
- Rapidly changing knowledge and citations make RAG the better first step
- Representative evaluation and rollback checkpoints cannot be prepared
Pretraining & CPT: system plate
- 01
Stage 1
Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus.
- 02
Stage 2
Treat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs.
- 03
Stage 3
Pin distributed execution and checkpoints to reproducible configurations with stage-specific stop gates.
- 04
Stage 4
Repeat acceptance evaluation after merge, quantization, and serving on the same representative tasks.
- The workflow begins with Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus..
- It reaches an acceptance decision through Domain and general capability regression.
03
Design method
Continued training controls what a model learns without silently sacrificing what it already knows.
- 01
Stage 1
Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus.
- 02
Stage 2
Treat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs.
- 03
Stage 3
Pin distributed execution and checkpoints to reproducible configurations with stage-specific stop gates.
- 04
Stage 4
Repeat acceptance evaluation after merge, quantization, and serving on the same representative tasks.
04
Application scenarios
A persistent domain-language and knowledge gap is not resolved by retrieval alone
CPT is not automatically justified when a general model struggles with organizational language. Retrieval, SFT, and prompting must be ruled in or out first.
- APPROACH
- Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus.
- BOUNDARY
- Training scope does not expand when corpus rights or lineage are unclear.
A long-term model strategy includes clear ownership and deployment boundaries
Small corpus biases can affect long-term knowledge, instruction following, and safety together. Discovering the wrong training direction late is costlier than a single run.
- APPROACH
- Treat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs.
- BOUNDARY
- Unpublished comparison scores and throughput without hardware conditions are not presented as outcomes.
05
Design choices
| Decision | Gain | Cost | Watch |
|---|---|---|---|
| A persistent domain-language and knowledge gap is not resolved by retrieval alone | Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus. | Training scope does not expand when corpus rights or lineage are unclear. | Domain and general capability regression |
| A long-term model strategy includes clear ownership and deployment boundaries | Treat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs. | Unpublished comparison scores and throughput without hardware conditions are not presented as outcomes. | Token efficiency and training stability |
Pretraining & CPT: system plate
| Record | Method | Acceptance evidence |
|---|---|---|
| R-1 | Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus. | Domain and general capability regression |
| R-2 | Treat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs. | Token efficiency and training stability |
| R-3 | Pin distributed execution and checkpoints to reproducible configurations with stage-specific stop gates. | Acceptance in the deployed form |
- The workflow begins with Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus..
- It reaches an acceptance decision through Domain and general capability regression.
07
Validation plan
| Measure | Method | Pass condition | Caveat |
|---|---|---|---|
| Domain and general capability regression | Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus. | Repeated runs satisfy the acceptance threshold agreed during discovery | Training scope does not expand when corpus rights or lineage are unclear. |
| Token efficiency and training stability | Treat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
| Acceptance in the deployed form | Pin distributed execution and checkpoints to reproducible configurations with stage-specific stop gates. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
08
Constraints and failure conditions
Rapidly changing knowledge and citations make RAG the better first step
Training scope does not expand when corpus rights or lineage are unclear.
Representative evaluation and rollback checkpoints cannot be prepared
Unpublished comparison scores and throughput without hardware conditions are not presented as outcomes.
09
Engagement model
- 01
Diagnosis
PattyAnalyze the current system and its failure signals.
ClientProvide representative work, data boundaries, and operating constraints.
Domain and general capability regression - 02
Design
PattyTreat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs.
ClientConfirm owners and acceptance criteria.
Token efficiency and training stability - 03
Validation
PattyPin distributed execution and checkpoints to reproducible configurations with stage-specific stop gates.
ClientMake the production-transition or stop decision.
Acceptance in the deployed form
10
Durable deliverables
- Pretraining & CPT decision record
- Continued training controls what a model learns without silently sacrificing what it already knows.Client-owned · Patty-reviewed
- Validation harness and acceptance criteria
- Domain and general capability regression · Token efficiency and training stability · Acceptance in the deployed formJointly maintained
- Operations and recovery runbook
- Training scope does not expand when corpus rights or lineage are unclear. · Unpublished comparison scores and throughput without hardware conditions are not presented as outcomes.Operating-team owned
11
Terminology
- Pretraining & CPT
- A CPT discipline spanning corpus diagnosis, mixing, tokenization, distributed runs, checkpoints, and serving acceptance.
- Acceptance criterion
- Domain and general capability regression
- Operating boundary
- Training scope does not expand when corpus rights or lineage are unclear.
REFERENCES
References and primary material
- Hugging Face language modeling
Primary material for the method and terminology.
- PEFT LoRA
Primary material for the method and terminology.