KV Cache Architecture
The KV cache is service state governing routing, isolation, reuse, and recovery—not merely a memory optimization.
Design block lifecycle, prefix reuse, tiered storage, eviction, and tenant boundaries around the workload.
§ 01
The operating conditions that justify KV Cache Architecture
As context and concurrency grow, cache fragmentation and eviction can bottleneck before compute.
Routing for reuse may worsen tenant isolation and tail latency.
- Long prefixes or system prompts repeat across requests
- Memory pressure and recomputation are material under long-context concurrency
- Requests are mostly short and non-repeating
- Cache keys and tenant isolation policy cannot be defined
KV Cache Architecture: system plate
- 01
State 1
Trace per-request cache occupancy, prefix duplication, and lifetime distributions.
- 02
State 2
Define an explicit lifecycle for block allocation, reference, replication, eviction, and recovery.
- 03
State 3
Measure bandwidth and recomputation break-even points across GPU, host, and remote tiers.
- 04
State 4
Validate isolation with failures including cache poisoning and cross-tenant leakage.
FEEDBACKFailed acceptance returns evidence to the first controlled stage: Prefix hit rate and recompute reduction.
- The workflow begins with Trace per-request cache occupancy, prefix duplication, and lifetime distributions..
- It reaches an acceptance decision through Prefix hit rate and recompute reduction.
§ 03
Fix the boundary and acceptance criteria before implementation.
The KV cache is service state governing routing, isolation, reuse, and recovery—not merely a memory optimization.
- 01
Stage 1
Trace per-request cache occupancy, prefix duplication, and lifetime distributions.
Review artifact 1 - 02
Stage 2
Define an explicit lifecycle for block allocation, reference, replication, eviction, and recovery.
Review artifact 2 - 03
Stage 3
Measure bandwidth and recomputation break-even points across GPU, host, and remote tiers.
Review artifact 3 - 04
Stage 4
Validate isolation with failures including cache poisoning and cross-tenant leakage.
Review artifact 4
§ 04
Hypothetical workloads make the applicability boundary concrete.
Long prefixes or system prompts repeat across requests
As context and concurrency grow, cache fragmentation and eviction can bottleneck before compute.
- APPROACH
- Trace per-request cache occupancy, prefix duplication, and lifetime distributions.
- BOUNDARY
- A higher hit rate does not always mean lower end-to-end latency.
Memory pressure and recomputation are material under long-context concurrency
Routing for reuse may worsen tenant isolation and tail latency.
- APPROACH
- Define an explicit lifecycle for block allocation, reference, replication, eviction, and recovery.
- BOUNDARY
- Sensitive prefixes prioritize isolation over reuse.
§ 05
Review gains and costs in the same table.
| Decision | Gain | Cost | Watch |
|---|---|---|---|
| Long prefixes or system prompts repeat across requests | Trace per-request cache occupancy, prefix duplication, and lifetime distributions. | A higher hit rate does not always mean lower end-to-end latency. | Prefix hit rate and recompute reduction |
| Memory pressure and recomputation are material under long-context concurrency | Define an explicit lifecycle for block allocation, reference, replication, eviction, and recovery. | Sensitive prefixes prioritize isolation over reuse. | Cache occupancy, eviction, and fragmentation |
KV Cache Architecture: system plate
- As context and concurrency grow, cache fragmentation and eviction can bottleneck before compute.
- A higher hit rate does not always mean lower end-to-end latency.
- Define an explicit lifecycle for block allocation, reference, replication, eviction, and recovery.
- Prefix hit rate and recompute reduction
- Tail latency and isolation
- Sensitive prefixes prioritize isolation over reuse.
- The workflow begins with Trace per-request cache occupancy, prefix duplication, and lifetime distributions..
- It reaches an acceptance decision through Prefix hit rate and recompute reduction.
§ 07
Agree on measurement conditions before publishing a result.
| Measure | Method | Pass condition | Caveat |
|---|---|---|---|
| Prefix hit rate and recompute reduction | Trace per-request cache occupancy, prefix duplication, and lifetime distributions. | Repeated runs satisfy the acceptance threshold agreed during discovery | A higher hit rate does not always mean lower end-to-end latency. |
| Cache occupancy, eviction, and fragmentation | Define an explicit lifecycle for block allocation, reference, replication, eviction, and recovery. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
| Tail latency and isolation | Measure bandwidth and recomputation break-even points across GPU, host, and remote tiers. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
§ 08
Conditions for not applying the capability are part of the design.
Requests are mostly short and non-repeating
A higher hit rate does not always mean lower end-to-end latency.
Cache keys and tenant isolation policy cannot be defined
Sensitive prefixes prioritize isolation over reuse.
§ 09
Proceed through diagnosis, design, and validation gates.
- 01
Diagnosis
PattyAnalyze the current system and its failure signals.
ClientProvide representative work, data boundaries, and operating constraints.
Prefix hit rate and recompute reduction - 02
Design
PattyDefine an explicit lifecycle for block allocation, reference, replication, eviction, and recovery.
ClientConfirm owners and acceptance criteria.
Cache occupancy, eviction, and fragmentation - 03
Validation
PattyMeasure bandwidth and recomputation break-even points across GPU, host, and remote tiers.
ClientMake the production-transition or stop decision.
Tail latency and isolation
§ 10
Artifacts remain with the operating organization after the engagement.
- KV Cache Architecture decision record
- The KV cache is service state governing routing, isolation, reuse, and recovery—not merely a memory optimization.Client-owned · Patty-reviewed
- Validation harness and acceptance criteria
- Prefix hit rate and recompute reduction · Cache occupancy, eviction, and fragmentation · Tail latency and isolationJointly maintained
- Operations and recovery runbook
- A higher hit rate does not always mean lower end-to-end latency. · Sensitive prefixes prioritize isolation over reuse.Operating-team owned
§ 11
Use shared terms with explicit operating meaning.
- KV Cache Architecture
- Design block lifecycle, prefix reuse, tiered storage, eviction, and tenant boundaries around the workload.
- Acceptance criterion
- Prefix hit rate and recompute reduction
- Operating boundary
- A higher hit rate does not always mean lower end-to-end latency.
REFERENCES
References and primary material
- PagedAttention
Primary material for the method and terminology.
- vLLM Automatic Prefix Caching
Primary material for the method and terminology.
Begin by determining whether KV Cache Architecture is the justified next step.
We define scope and validation against representative work, data and infrastructure boundaries, and explicit failure conditions.