Temporal State Compression: An Audit-Driven Study of Transformer Runtime State
From an overstated KV-cache headline to a stricter accounting model, a fail-closed benchmark harness, and a narrower systems question about compressed state and cross-request reuse.
Abstract
Temporal State Compression (TSC) began with a simple idea: transformer KV state evolves over time, so perhaps it should be represented as related state rather than repeatedly materialized as independent full snapshots.
Early experiments produced large compression ratios under cumulative temporal-history accounting. A later end-to-end audit showed that this was being interpreted too broadly: a ratio measured against repeated intermediate states is not the same thing as compressing the single current KV snapshot required for the next decode step. Under the audited final-snapshot path, frequent keyframe fallback produced a 0.5× result, meaning the encoded representation was approximately twice as large as the final FP16 snapshot.
The research then split into three separate questions: temporal-history representation, isolated KV-snapshot compression, and serving-level state reuse. A later benchmark harness, TSC Bench, added exact byte accounting and fixed-width compression ceilings. That harness identified a second claim problem: previously reported no-eviction ratios of 4.1× K4V4 and 6.1–6.2× K4V2 exceed the arithmetic ceilings of fixed-width representations unless additional measured serialization behavior explains them. Those historical figures are therefore withheld from the current submission claim set pending re-measurement.
1. Three Different State Boundaries
The project originally moved across three related but non-interchangeable quantities. The audit made the boundaries explicit.
How compactly can a sequence of related runtime states be retained or reconstructed?
How compactly can the one KV state needed for the next decode step be represented?
How much state is duplicated across concurrent sessions, shared prefixes, and divergent branches?
The core mistake in the original public framing was not fabricating a measurement. It was transferring a valid measurement from one denominator into a stronger interpretation governed by another denominator.
2. Audit Trail
Large temporal-history ratios
Early GPT-2 experiments reported headline ratios as high as 63×. Those results are retained as historical temporal-history measurements, not as evidence of 63× single-snapshot KV compression.
The 0.5× result that changed the project
With a final FP16 snapshot baseline and the actual TSC storage path, the default configuration fell back to full keyframes on roughly 97–99% of steps. The encoded representation was about twice the size of the final state. The single-snapshot interpretation failed.
Snapshot compression and state reuse separated
The work split into isolated KV representation, token retention, and cross-request reuse. Reuse was explicitly treated as a serving/workload property rather than a stronger codec ratio.
The harness challenged the corrected manuscript too
TSC Bench added exact packed-bit accounting, metadata accounting, reproducibility manifests, and fixed-width compression ceilings. It showed that K4V4 cannot exceed 4.0× and K4V2 cannot exceed 5.333× before metadata without an actual serialized mechanism that compresses below the nominal bit width.
Historical 4.1× / 6.2× no-eviction figures are on hold
Those figures remain part of the research history, but they are not accepted as current submission results until they are reproduced through the audited benchmark path. The paper-grade result gate remains closed while the required Mistral-7B LongBench E5 run is pending.
3. Current Claim Boundary
| Claim | Status | Current wording |
|---|---|---|
| 63× single-snapshot KV compression | Retired | The 63× result belonged to temporal-history accounting and should not be used as a final-snapshot production claim. |
| 4.1× K4V4 no-eviction | Re-measure | Historical scale-validation figure. It exceeds the fixed-width 4.0× ceiling before metadata and must be reproduced from actual serialized bytes before submission. |
| 6.1–6.2× K4V2 no-eviction | Re-measure | Historical scale-validation figure. Fixed-width K4V2 has a 5.333× ceiling before metadata, so the old result is withheld pending audited reproduction. |
| Cross-request reuse can increase effective memory savings | Analytical only | Reuse can multiply effective fleet-level savings when state is genuinely shared, but the multiplier belongs to the workload and serving layer, not to the codec. |
| Serving throughput / HBM improvement | Not yet established | Requires a production-like compressed-attention path and a matched concurrent serving benchmark. |
| Environmental / fleet-cost savings | Withheld | Not restored until fixed-SLA GPU-count and wall-power reductions are measured end to end. |
4. TSC Bench: The Evidence Harness
TSC Bench was built after the manuscript audit to make it harder for the same class of claim error to recur. Its purpose is narrower than the original research pages: generate evidence that can survive an ML-systems review.
Counts code bits plus scale/zero-point metadata instead of inferring compression from tensor dtype or nominal bit width.
Flags impossible no-eviction ratios unless a measured serialized bitstream explains how nominal bit width was beaten.
Run manifests record exact config, environment, CUDA/Torch/GPU metadata, and git commit.
Deterministic mock runs validate the measurement pipeline but are explicitly rejected as paper-grade evidence.
Quantization-only experiments extract real model KV state, compress/decompress it, reinject it, and compare against the untouched cache.
The harness fails fast because generic cache reinjection does not preserve arbitrary per-layer token positions reliably enough for a valid eviction-quality claim.
The load generator and GPU telemetry path exist, but compressed cache must remain compressed through attention before a serious serving claim is allowed.
The public paper gate remains closed until the required Mistral-7B LongBench experiment is executed and audited.
5. What Is Still Missing
The current work is deliberately not presented as finished. The immediate paper-grade gate is the Mistral-7B LongBench E5 run. More broadly, a competitive ML-systems claim requires evidence beyond local next-token distortion.
- Paper-grade downstream evaluation. LongBench first, followed by broader long-context and shared-context task coverage.
- Matched baselines. Same model, dataset, context lengths, hardware, and serving stack for native KV and strong compression/eviction baselines.
- Production-like compressed attention. The compressed cache must remain compressed through the attention path; whole-cache dequantization is not a serious serving result.
- Concurrent serving measurements. TTFT, inter-token latency, throughput, requests/s, GPU-resident bytes, utilization, and reuse hit rates under replayable load.
- Reuse validation. Compare measured shared-token fractions and fleet memory against the analytical reuse model instead of presenting reuse scenarios as codec ratios.
6. Questions I Would Most Value From a Reviewer
Please try to break the measurement, not just the prose.
I am especially interested in criticism of the following:
- Is the unit of analysis correct for each claimed ratio?
- Is any denominator still counting duplicated state in a way that changes the interpretation?
- Does the serialized-byte accounting miss metadata, padding, alignment, or temporary workspace?
- Are any reuse assumptions being smuggled into a codec result?
- Are the quality metrics sufficient for the claim being made, or is a downstream task required?
- What matched baseline would make the current experiment scientifically uninteresting if TSC fails to beat it?
- Which result should be removed entirely rather than merely caveated?
7. Evidence Ledger
This page is a review-facing synthesis, not a replacement for the underlying artifacts. The governing sources are listed by role so changes in claim status can be traced.