SFT Data Pipelines
SFT quality is decided by data contracts, lineage, splits, and release gates before the trainer runs.
Design source ingestion, schemas, deduplication, contamination control, human audit, calibration, and immutable exports.
01
Problem definition
Collecting examples that look good cannot explain the origin of errors and bias.
When training, evaluation, and quantization calibration data mix, causes of performance changes cannot be reproduced.
- Repeatable instruction behavior must be internalized in the model
- Dataset and model versions must remain traceable over time
- The core problem is access to current knowledge
- Answer schemas and human-audit criteria are not agreed
SFT Data Pipelines: system plate
- 01
Stage 1
Define behavior-level schemas and required metadata before collection.
- 02
Stage 2
Run exact, near-duplicate, and evaluation-contamination checks as separate passes with rejection reasons.
- 03
Stage 3
Feed human-audit samples and feedback back into rules for the next export.
- 04
Stage 4
Bind train, evaluation, and calibration splits to an immutable manifest linked to the model.
- The workflow begins with Define behavior-level schemas and required metadata before collection..
- It reaches an acceptance decision through Schema pass and rejection distribution.
03
Design method
SFT quality is decided by data contracts, lineage, splits, and release gates before the trainer runs.
- 01
Stage 1
Define behavior-level schemas and required metadata before collection.
- 02
Stage 2
Run exact, near-duplicate, and evaluation-contamination checks as separate passes with rejection reasons.
- 03
Stage 3
Feed human-audit samples and feedback back into rules for the next export.
- 04
Stage 4
Bind train, evaluation, and calibration splits to an immutable manifest linked to the model.
04
Application scenarios
Repeatable instruction behavior must be internalized in the model
Collecting examples that look good cannot explain the origin of errors and bias.
- APPROACH
- Define behavior-level schemas and required metadata before collection.
- BOUNDARY
- Generation volume is not used as evidence of quality.
Dataset and model versions must remain traceable over time
When training, evaluation, and quantization calibration data mix, causes of performance changes cannot be reproduced.
- APPROACH
- Run exact, near-duplicate, and evaluation-contamination checks as separate passes with rejection reasons.
- BOUNDARY
- A release does not rely on automated scores without human audit.
05
Design choices
| Decision | Gain | Cost | Watch |
|---|---|---|---|
| Repeatable instruction behavior must be internalized in the model | Define behavior-level schemas and required metadata before collection. | Generation volume is not used as evidence of quality. | Schema pass and rejection distribution |
| Dataset and model versions must remain traceable over time | Run exact, near-duplicate, and evaluation-contamination checks as separate passes with rejection reasons. | A release does not rely on automated scores without human audit. | Deduplication and contamination results |
SFT Data Pipelines: system plate
| Item | Method | Evidence | Boundary |
|---|---|---|---|
| Layer 1 | Define behavior-level schemas and required metadata before collection. | Schema pass and rejection distribution | Generation volume is not used as evidence of quality. |
| Layer 2 | Run exact, near-duplicate, and evaluation-contamination checks as separate passes with rejection reasons. | Deduplication and contamination results | A release does not rely on automated scores without human audit. |
| Layer 3 | Feed human-audit samples and feedback back into rules for the next export. | Auditor agreement and regression | Generation volume is not used as evidence of quality. |
- The workflow begins with Define behavior-level schemas and required metadata before collection..
- It reaches an acceptance decision through Schema pass and rejection distribution.
07
Validation plan
| Measure | Method | Pass condition | Caveat |
|---|---|---|---|
| Schema pass and rejection distribution | Define behavior-level schemas and required metadata before collection. | Repeated runs satisfy the acceptance threshold agreed during discovery | Generation volume is not used as evidence of quality. |
| Deduplication and contamination results | Run exact, near-duplicate, and evaluation-contamination checks as separate passes with rejection reasons. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
| Auditor agreement and regression | Feed human-audit samples and feedback back into rules for the next export. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
08
Constraints and failure conditions
The core problem is access to current knowledge
Generation volume is not used as evidence of quality.
Answer schemas and human-audit criteria are not agreed
A release does not rely on automated scores without human audit.
09
Engagement model
- 01
Diagnosis
PattyAnalyze the current system and its failure signals.
ClientProvide representative work, data boundaries, and operating constraints.
Schema pass and rejection distribution - 02
Design
PattyRun exact, near-duplicate, and evaluation-contamination checks as separate passes with rejection reasons.
ClientConfirm owners and acceptance criteria.
Deduplication and contamination results - 03
Validation
PattyFeed human-audit samples and feedback back into rules for the next export.
ClientMake the production-transition or stop decision.
Auditor agreement and regression
10
Durable deliverables
- SFT Data Pipelines decision record
- SFT quality is decided by data contracts, lineage, splits, and release gates before the trainer runs.Client-owned · Patty-reviewed
- Validation harness and acceptance criteria
- Schema pass and rejection distribution · Deduplication and contamination results · Auditor agreement and regressionJointly maintained
- Operations and recovery runbook
- Generation volume is not used as evidence of quality. · A release does not rely on automated scores without human audit.Operating-team owned
11
Terminology
- SFT Data Pipelines
- Design source ingestion, schemas, deduplication, contamination control, human audit, calibration, and immutable exports.
- Acceptance criterion
- Schema pass and rejection distribution
- Operating boundary
- Generation volume is not used as evidence of quality.
REFERENCES
References and primary material
- TRL SFT Trainer
Primary material for the method and terminology.
- Hugging Face Datasets
Primary material for the method and terminology.