Model Evaluation & Red Teaming
Evaluation must cover average quality and severe failures, then connect findings to deployment decisions.
Manage capability, safety, security, bias, and tool-use failures through threat models and evidence-based release gates.
§ 01
The operating conditions that justify Model Evaluation & Red Teaming
Average-case testing misses rare but consequential failures and adversarial paths.
Red-team findings without owners and closure criteria become reports while risk reaches production.
- A model touches sensitive data, external tools, or customer decisions
- Risk regression must be tested for every model or policy update
- No threat model or risk-acceptance owner exists
- There is no authority to remediate or block findings
Model Evaluation & Red Teaming: system plate
- 01
State 1
Begin with a threat model connecting assets, actors, permissions, and harm paths.
- 02
State 2
Combine automated attack generation with domain-expert manual exploration.
- 03
State 3
Maintain a finding ledger with reproducible inputs, environment, impact, and mitigation.
- 04
State 4
Do not close a release gate without retest evidence and residual-risk acceptance.
FEEDBACKFailed acceptance returns evidence to the first controlled stage: Threat coverage and attack surface.
- The workflow begins with Begin with a threat model connecting assets, actors, permissions, and harm paths..
- It reaches an acceptance decision through Threat coverage and attack surface.
§ 03
Fix the boundary and acceptance criteria before implementation.
Evaluation must cover average quality and severe failures, then connect findings to deployment decisions.
- 01
Stage 1
Begin with a threat model connecting assets, actors, permissions, and harm paths.
Review artifact 1 - 02
Stage 2
Combine automated attack generation with domain-expert manual exploration.
Review artifact 2 - 03
Stage 3
Maintain a finding ledger with reproducible inputs, environment, impact, and mitigation.
Review artifact 3 - 04
Stage 4
Do not close a release gate without retest evidence and residual-risk acceptance.
Review artifact 4
§ 04
Hypothetical workloads make the applicability boundary concrete.
A model touches sensitive data, external tools, or customer decisions
Average-case testing misses rare but consequential failures and adversarial paths.
- APPROACH
- Begin with a threat model connecting assets, actors, permissions, and harm paths.
- BOUNDARY
- Red teaming does not prove safety; it expands the known failure surface.
Risk regression must be tested for every model or policy update
Red-team findings without owners and closure criteria become reports while risk reaches production.
- APPROACH
- Combine automated attack generation with domain-expert manual exploration.
- BOUNDARY
- Attack success in a test environment is not generalized to all production conditions.
§ 05
Review gains and costs in the same table.
| Decision | Gain | Cost | Watch |
|---|---|---|---|
| A model touches sensitive data, external tools, or customer decisions | Begin with a threat model connecting assets, actors, permissions, and harm paths. | Red teaming does not prove safety; it expands the known failure surface. | Threat coverage and attack surface |
| Risk regression must be tested for every model or policy update | Combine automated attack generation with domain-expert manual exploration. | Attack success in a test environment is not generalized to all production conditions. | Finding reproduction and remediation rate |
Model Evaluation & Red Teaming: system plate
- No threat model or risk-acceptance owner exists
Red teaming does not prove safety; it expands the known failure surface.
- SIGNAL
- Threat coverage and attack surface
- MITIGATION
- Combine automated attack generation with domain-expert manual exploration.
- There is no authority to remediate or block findings
Attack success in a test environment is not generalized to all production conditions.
- SIGNAL
- Finding reproduction and remediation rate
- MITIGATION
- Maintain a finding ledger with reproducible inputs, environment, impact, and mitigation.
- The workflow begins with Begin with a threat model connecting assets, actors, permissions, and harm paths..
- It reaches an acceptance decision through Threat coverage and attack surface.
§ 07
Agree on measurement conditions before publishing a result.
| Measure | Method | Pass condition | Caveat |
|---|---|---|---|
| Threat coverage and attack surface | Begin with a threat model connecting assets, actors, permissions, and harm paths. | Repeated runs satisfy the acceptance threshold agreed during discovery | Red teaming does not prove safety; it expands the known failure surface. |
| Finding reproduction and remediation rate | Combine automated attack generation with domain-expert manual exploration. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
| Residual-risk approval traceability | Maintain a finding ledger with reproducible inputs, environment, impact, and mitigation. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
§ 08
Conditions for not applying the capability are part of the design.
No threat model or risk-acceptance owner exists
Red teaming does not prove safety; it expands the known failure surface.
There is no authority to remediate or block findings
Attack success in a test environment is not generalized to all production conditions.
§ 09
Proceed through diagnosis, design, and validation gates.
- 01
Diagnosis
PattyAnalyze the current system and its failure signals.
ClientProvide representative work, data boundaries, and operating constraints.
Threat coverage and attack surface - 02
Design
PattyCombine automated attack generation with domain-expert manual exploration.
ClientConfirm owners and acceptance criteria.
Finding reproduction and remediation rate - 03
Validation
PattyMaintain a finding ledger with reproducible inputs, environment, impact, and mitigation.
ClientMake the production-transition or stop decision.
Residual-risk approval traceability
§ 10
Artifacts remain with the operating organization after the engagement.
- Model Evaluation & Red Teaming decision record
- Evaluation must cover average quality and severe failures, then connect findings to deployment decisions.Client-owned · Patty-reviewed
- Validation harness and acceptance criteria
- Threat coverage and attack surface · Finding reproduction and remediation rate · Residual-risk approval traceabilityJointly maintained
- Operations and recovery runbook
- Red teaming does not prove safety; it expands the known failure surface. · Attack success in a test environment is not generalized to all production conditions.Operating-team owned
§ 11
Use shared terms with explicit operating meaning.
- Model Evaluation & Red Teaming
- Manage capability, safety, security, bias, and tool-use failures through threat models and evidence-based release gates.
- Acceptance criterion
- Threat coverage and attack surface
- Operating boundary
- Red teaming does not prove safety; it expands the known failure surface.
REFERENCES
References and primary material
- NIST AI 600-1
Primary material for the method and terminology.
- MITRE ATLAS
Primary material for the method and terminology.
Begin by determining whether Model Evaluation & Red Teaming is the justified next step.
We define scope and validation against representative work, data and infrastructure boundaries, and explicit failure conditions.