Pretraining & CPT
Continued training controls what a model learns without silently sacrificing what it already knows.
A CPT discipline spanning corpus diagnosis, mixing, tokenization, distributed runs, checkpoints, and serving acceptance.
§ 01
The operating conditions that justify Pretraining & CPT
CPT is not automatically justified when a general model struggles with organizational language. Retrieval, SFT, and prompting must be ruled in or out first.
Small corpus biases can affect long-term knowledge, instruction following, and safety together. Discovering the wrong training direction late is costlier than a single run.
- A persistent domain-language and knowledge gap is not resolved by retrieval alone
- A long-term model strategy includes clear ownership and deployment boundaries
- Rapidly changing knowledge and citations make RAG the better first step
- Representative evaluation and rollback checkpoints cannot be prepared
Pretraining & CPT: system plate
- 01
Stage 1
Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus.
- 02
Stage 2
Treat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs.
- 03
Stage 3
Pin distributed execution and checkpoints to reproducible configurations with stage-specific stop gates.
- 04
Stage 4
Repeat acceptance evaluation after merge, quantization, and serving on the same representative tasks.
- The workflow begins with Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus..
- It reaches an acceptance decision through Domain and general capability regression.
§ 03
Fix the boundary and acceptance criteria before implementation.
Continued training controls what a model learns without silently sacrificing what it already knows.
- 01
Stage 1
Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus.
Review artifact 1 - 02
Stage 2
Treat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs.
Review artifact 2 - 03
Stage 3
Pin distributed execution and checkpoints to reproducible configurations with stage-specific stop gates.
Review artifact 3 - 04
Stage 4
Repeat acceptance evaluation after merge, quantization, and serving on the same representative tasks.
Review artifact 4
§ 04
Hypothetical workloads make the applicability boundary concrete.
A persistent domain-language and knowledge gap is not resolved by retrieval alone
CPT is not automatically justified when a general model struggles with organizational language. Retrieval, SFT, and prompting must be ruled in or out first.
- APPROACH
- Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus.
- BOUNDARY
- Training scope does not expand when corpus rights or lineage are unclear.
A long-term model strategy includes clear ownership and deployment boundaries
Small corpus biases can affect long-term knowledge, instruction following, and safety together. Discovering the wrong training direction late is costlier than a single run.
- APPROACH
- Treat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs.
- BOUNDARY
- Unpublished comparison scores and throughput without hardware conditions are not presented as outcomes.
§ 05
Review gains and costs in the same table.
| Decision | Gain | Cost | Watch |
|---|---|---|---|
| A persistent domain-language and knowledge gap is not resolved by retrieval alone | Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus. | Training scope does not expand when corpus rights or lineage are unclear. | Domain and general capability regression |
| A long-term model strategy includes clear ownership and deployment boundaries | Treat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs. | Unpublished comparison scores and throughput without hardware conditions are not presented as outcomes. | Token efficiency and training stability |
Pretraining & CPT: system plate
| Record | Method | Acceptance evidence |
|---|---|---|
| R-1 | Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus. | Domain and general capability regression |
| R-2 | Treat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs. | Token efficiency and training stability |
| R-3 | Pin distributed execution and checkpoints to reproducible configurations with stage-specific stop gates. | Acceptance in the deployed form |
- The workflow begins with Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus..
- It reaches an acceptance decision through Domain and general capability regression.
§ 07
Agree on measurement conditions before publishing a result.
| Measure | Method | Pass condition | Caveat |
|---|---|---|---|
| Domain and general capability regression | Diagnose duplication, time range, rights, and language distribution before fixing the trainable corpus. | Repeated runs satisfy the acceptance threshold agreed during discovery | Training scope does not expand when corpus rights or lineage are unclear. |
| Token efficiency and training stability | Treat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
| Acceptance in the deployed form | Pin distributed execution and checkpoints to reproducible configurations with stage-specific stop gates. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
§ 08
Conditions for not applying the capability are part of the design.
Rapidly changing knowledge and citations make RAG the better first step
Training scope does not expand when corpus rights or lineage are unclear.
Representative evaluation and rollback checkpoints cannot be prepared
Unpublished comparison scores and throughput without hardware conditions are not presented as outcomes.
§ 09
Proceed through diagnosis, design, and validation gates.
- 01
Diagnosis
PattyAnalyze the current system and its failure signals.
ClientProvide representative work, data boundaries, and operating constraints.
Domain and general capability regression - 02
Design
PattyTreat domain, general, and replay ratios as hypotheses and compare forgetting and loss in small runs.
ClientConfirm owners and acceptance criteria.
Token efficiency and training stability - 03
Validation
PattyPin distributed execution and checkpoints to reproducible configurations with stage-specific stop gates.
ClientMake the production-transition or stop decision.
Acceptance in the deployed form
§ 10
Artifacts remain with the operating organization after the engagement.
- Pretraining & CPT decision record
- Continued training controls what a model learns without silently sacrificing what it already knows.Client-owned · Patty-reviewed
- Validation harness and acceptance criteria
- Domain and general capability regression · Token efficiency and training stability · Acceptance in the deployed formJointly maintained
- Operations and recovery runbook
- Training scope does not expand when corpus rights or lineage are unclear. · Unpublished comparison scores and throughput without hardware conditions are not presented as outcomes.Operating-team owned
§ 11
Use shared terms with explicit operating meaning.
- Pretraining & CPT
- A CPT discipline spanning corpus diagnosis, mixing, tokenization, distributed runs, checkpoints, and serving acceptance.
- Acceptance criterion
- Domain and general capability regression
- Operating boundary
- Training scope does not expand when corpus rights or lineage are unclear.
REFERENCES
References and primary material
- Hugging Face language modeling
Primary material for the method and terminology.
- PEFT LoRA
Primary material for the method and terminology.
Begin by determining whether Pretraining & CPT is the justified next step.
We define scope and validation against representative work, data and infrastructure boundaries, and explicit failure conditions.