Model Evaluation & Red Teaming
Evaluation must cover average quality and severe failures, then connect findings to deployment decisions.
Manage capability, safety, security, bias, and tool-use failures through threat models and evidence-based release gates.
01
Problem definition
Average-case testing misses rare but consequential failures and adversarial paths.
Red-team findings without owners and closure criteria become reports while risk reaches production.
- A model touches sensitive data, external tools, or customer decisions
- Risk regression must be tested for every model or policy update
- No threat model or risk-acceptance owner exists
- There is no authority to remediate or block findings
Model Evaluation & Red Teaming: system plate
- 01
State 1
Begin with a threat model connecting assets, actors, permissions, and harm paths.
- 02
State 2
Combine automated attack generation with domain-expert manual exploration.
- 03
State 3
Maintain a finding ledger with reproducible inputs, environment, impact, and mitigation.
- 04
State 4
Do not close a release gate without retest evidence and residual-risk acceptance.
FEEDBACKFailed acceptance returns evidence to the first controlled stage: Threat coverage and attack surface.
- The workflow begins with Begin with a threat model connecting assets, actors, permissions, and harm paths..
- It reaches an acceptance decision through Threat coverage and attack surface.
03
Design method
Evaluation must cover average quality and severe failures, then connect findings to deployment decisions.
- 01
Stage 1
Begin with a threat model connecting assets, actors, permissions, and harm paths.
- 02
Stage 2
Combine automated attack generation with domain-expert manual exploration.
- 03
Stage 3
Maintain a finding ledger with reproducible inputs, environment, impact, and mitigation.
- 04
Stage 4
Do not close a release gate without retest evidence and residual-risk acceptance.
04
Application scenarios
A model touches sensitive data, external tools, or customer decisions
Average-case testing misses rare but consequential failures and adversarial paths.
- APPROACH
- Begin with a threat model connecting assets, actors, permissions, and harm paths.
- BOUNDARY
- Red teaming does not prove safety; it expands the known failure surface.
Risk regression must be tested for every model or policy update
Red-team findings without owners and closure criteria become reports while risk reaches production.
- APPROACH
- Combine automated attack generation with domain-expert manual exploration.
- BOUNDARY
- Attack success in a test environment is not generalized to all production conditions.
05
Design choices
| Decision | Gain | Cost | Watch |
|---|---|---|---|
| A model touches sensitive data, external tools, or customer decisions | Begin with a threat model connecting assets, actors, permissions, and harm paths. | Red teaming does not prove safety; it expands the known failure surface. | Threat coverage and attack surface |
| Risk regression must be tested for every model or policy update | Combine automated attack generation with domain-expert manual exploration. | Attack success in a test environment is not generalized to all production conditions. | Finding reproduction and remediation rate |
Model Evaluation & Red Teaming: system plate
- No threat model or risk-acceptance owner exists
Red teaming does not prove safety; it expands the known failure surface.
- SIGNAL
- Threat coverage and attack surface
- MITIGATION
- Combine automated attack generation with domain-expert manual exploration.
- There is no authority to remediate or block findings
Attack success in a test environment is not generalized to all production conditions.
- SIGNAL
- Finding reproduction and remediation rate
- MITIGATION
- Maintain a finding ledger with reproducible inputs, environment, impact, and mitigation.
- The workflow begins with Begin with a threat model connecting assets, actors, permissions, and harm paths..
- It reaches an acceptance decision through Threat coverage and attack surface.
07
Validation plan
| Measure | Method | Pass condition | Caveat |
|---|---|---|---|
| Threat coverage and attack surface | Begin with a threat model connecting assets, actors, permissions, and harm paths. | Repeated runs satisfy the acceptance threshold agreed during discovery | Red teaming does not prove safety; it expands the known failure surface. |
| Finding reproduction and remediation rate | Combine automated attack generation with domain-expert manual exploration. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
| Residual-risk approval traceability | Maintain a finding ledger with reproducible inputs, environment, impact, and mitigation. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
08
Constraints and failure conditions
No threat model or risk-acceptance owner exists
Red teaming does not prove safety; it expands the known failure surface.
There is no authority to remediate or block findings
Attack success in a test environment is not generalized to all production conditions.
09
Engagement model
- 01
Diagnosis
PattyAnalyze the current system and its failure signals.
ClientProvide representative work, data boundaries, and operating constraints.
Threat coverage and attack surface - 02
Design
PattyCombine automated attack generation with domain-expert manual exploration.
ClientConfirm owners and acceptance criteria.
Finding reproduction and remediation rate - 03
Validation
PattyMaintain a finding ledger with reproducible inputs, environment, impact, and mitigation.
ClientMake the production-transition or stop decision.
Residual-risk approval traceability
10
Durable deliverables
- Model Evaluation & Red Teaming decision record
- Evaluation must cover average quality and severe failures, then connect findings to deployment decisions.Client-owned · Patty-reviewed
- Validation harness and acceptance criteria
- Threat coverage and attack surface · Finding reproduction and remediation rate · Residual-risk approval traceabilityJointly maintained
- Operations and recovery runbook
- Red teaming does not prove safety; it expands the known failure surface. · Attack success in a test environment is not generalized to all production conditions.Operating-team owned
11
Terminology
- Model Evaluation & Red Teaming
- Manage capability, safety, security, bias, and tool-use failures through threat models and evidence-based release gates.
- Acceptance criterion
- Threat coverage and attack surface
- Operating boundary
- Red teaming does not prove safety; it expands the known failure surface.
REFERENCES
References and primary material
- NIST AI 600-1
Primary material for the method and terminology.
- MITRE ATLAS
Primary material for the method and terminology.