Scale Validation:
TSC Compression Holds from
7B to 72B Parameters

The Scale Question

TSC — Token-State Compression — combines attention-aware eviction with asymmetric KV quantization (4-bit keys, 2-bit values) to shrink the KV cache at runtime without modifying model weights. Prior work demonstrated 10.6× single-snapshot compression and 32–113× serving-level compression on TinyLlama (1.1B parameters).

But TinyLlama is a toy model. Production LLM serving runs Llama 70B, Mixtral 8x22B, Qwen2 72B, and their successors. The critical question: do compression ratios transfer across model scale, or do they degrade as parameter count climbs?

To answer this, we ran the complete TSC pipeline — all four tier configs plus eight extended sweep configs — on three GQA-architecture models spanning a 10× range in parameter count, on a single NVIDIA H100 80GB GPU.

Test Matrix

Core Tier Results: Ratios Are Architecture-Independent

The headline finding: compression ratios are effectively constant across model scale. Lossless holds at 4.1× whether you compress 7B or 72B. Conservative holds at 6.1–6.2×. This is expected — the ratios are determined by quantization arithmetic (FP16 → INT4/INT2) and the eviction keep-ratio, not by model size. But confirming it empirically on production-scale models eliminates the "maybe it breaks at scale" risk.

Quality at Scale: Larger Models Have More Redundancy

While ratios are constant, quality metrics tell a more nuanced story. KL divergence and Top-1 agreement vary by model — and the variation reveals something important about how model scale interacts with KV cache compression.

The 72B Paradox

Qwen2 72B shows better quality than the smaller models at Lossless and Conservative tiers — KL of 0.002 and perfect top-1 at Lossless, versus 0.003/0.967 on 7B. This is consistent with the hypothesis that larger models distribute information more broadly across KV heads, creating natural redundancy that quantization can exploit.

However, the 72B's Aggressive tier shows elevated KL (0.220, rated MARGINAL) compared to the 32B's 0.123 (ACCEPTABLE). The explanation: at aggressive eviction (keep only 10%), the 72B's 80 layers amplify error propagation. Early-layer eviction mistakes compound through more transformer blocks. This is not a failure of TSC — it's a well-understood property of deep networks that sets the practical floor for how aggressively you should evict.

Full Sweep: 12 Configurations, Two Production Models

Beyond the four core tiers, we ran eight additional configurations on both the 32B and 72B models to map the Pareto frontier between compression and quality.

Hidden Gem: K4V2 PE 50%

The sweep uncovered an interesting operating point that the four core tiers miss. K4V2 with 50% preserve-early eviction achieves 9.0× compression with KL 0.039 and top-1 0.980 on the 72B — nearly Conservative-tier quality at Balanced-tier compression. This may be the optimal production configuration for operators who want maximum compression without quality risk.

Adaptive vs. Preserve-Early Eviction

Two eviction schedules were compared: preserve-early (PE), which unconditionally keeps the first N tokens, and adaptive, which uses attention-weighted importance scores to choose which tokens to keep.

At matched ratios, adaptive shows a consistent ~0.01–0.03 KL advantage on the 72B (e.g., Adapt 30% gives KL 0.049 vs PE 30%'s 0.076). The advantage narrows at higher compression. For production, adaptive eviction is worth the extra compute if you're targeting the 9–10× range.

GQA Interaction: TSC Stacks on Top

All three benchmarked models use Grouped Query Attention (GQA), where multiple attention heads share fewer KV heads. This is already a form of KV compression — Qwen2 72B's 8:1 GQA ratio means it stores 8× fewer KV vectors than a hypothetical MHA model of the same width.

TSC compression is independent of and stacks on top of GQA. GQA reduces the number of KV heads. TSC quantizes and evicts within whatever heads exist. The two are orthogonal optimizations, and their compression ratios multiply:

For Qwen2 72B, the combined compression is: GQA (8×) × TSC Conservative (6.2×) = 49.6× total KV compression relative to an uncompressed MHA baseline. With serving-level dedup added, the effective multiplier exceeds 100×.

Serving-Level Projections

Single-snapshot compression is the building block. In a real serving deployment, DeltaStore's prefix deduplication, temporal delta compression, and cross-user sharing multiply the effective ratio dramatically. Here are the projections for a 100-user system prompt sharing scenario:

At Conservative tier with 100 concurrent users, a 72B model's KV cache drops from 16 GB (100 × 160 MB) to approximately 260 MB. That's the difference between needing 4 H100s for KV alone versus fitting everything on one.

The Production Sell: From 6.2× Snapshot to 62× Serving

A 6.2× compression ratio sounds good on paper. But it undersells the real-world impact by an order of magnitude. The reason is that single-snapshot compression is only the first layer of a three-layer serving optimization stack. Understanding how 6.2× becomes 62× is the key to understanding why TSC changes the economics of LLM deployment.

Layer 1: Snapshot Compression (6.2×)

This is what the benchmark measures directly. Every user's KV cache is independently compressed from FP16 to K4V2 (4-bit keys, 2-bit values with rotation). A single user's 160 MB cache for a 72B model at 4K context drops to 26 MB. No eviction, no token loss, GOOD-grade quality (KL 0.024, top-1 0.980).

Layer 2: Prefix Deduplication (6.2× → 19–31×)

In production, every user hitting the same API endpoint receives the same system prompt. A customer-support chatbot might have a 2,000-token system prompt shared across all sessions. Without TSC, that prefix is computed and stored independently for each user — 100 users means 100 copies of identical KV entries.

DeltaStore's prefix deduplication layer detects identical KV prefixes and stores them once. Users share a read-only reference to the compressed prefix; only the unique suffix (the user's actual conversation) is stored per-user.

The dedup multiplier depends on the prefix-to-suffix ratio. A longer system prompt (common in production — RAG context, tool definitions, persona instructions) means more sharing:

The 62× figure is not a theoretical maximum — it's what happens with a typical agentic workload where 87% of context is shared tool definitions, function schemas, and system instructions. This is the fastest-growing category of LLM deployment.

Layer 3: Temporal Delta Compression (62× → 90–113×)

The third layer exploits a property unique to multi-turn conversations: successive KV caches differ by only a few tokens. When a user sends their next message, the KV cache grows by only the new tokens — the previous turns' KV entries are unchanged.

DeltaStore's temporal compression stores only the delta between the previous and current cache state. For a conversation where each turn adds 50 tokens to a 4K context, the delta is 1.25% of the full cache. Combined with the other two layers:

Why This Is a Sell for Production

Consider a concrete deployment: a SaaS company running a 72B model as an AI assistant for their product, serving 1,000 concurrent users with a 4K context window.

The last row is the most important for product differentiation. Without KV compression, operators must choose between long context windows and concurrent users — they can't afford both. With TSC, the KV cache stops being the bottleneck, and the limit shifts from memory to compute. This unlocks product features (long conversations, RAG with large retrievals, multi-document analysis) that competitors without compression simply cannot offer at the same price point.

The Competitive Moat

Weight quantization (GPTQ, AWQ, GGUF) is commodity technology — everyone uses it, and the ratios are similar across vendors. KV cache compression at serving level is not. No major inference provider (vLLM, TGI, TensorRT-LLM) currently ships production-grade KV cache compression with cross-user dedup and temporal deltas.

An operator deploying TSC gets a structural cost advantage that compounds with scale:

Assumptions: 72B model, 4K context, 50% prefix sharing, Conservative tier (6.2× snapshot), H100 80GB at $2.50/hr on-demand. Real savings are higher with reserved instances and longer context windows.

What This Validates (and What It Doesn’t)

Validated

Not Yet Validated

Verdict

The benchmark eliminates the "toy model" objection. TSC's theoretical foundations — quantization arithmetic determines ratios, attention sparsity enables eviction, GQA stacks orthogonally — are now empirically confirmed across a 10× range in parameter count on production hardware.

What remains is engineering: end-to-end integration benchmarks, latency profiling under concurrent load, and task-specific evaluation. The compression science is settled.

Methodology & Reproducibility

All results are deterministic given fixed random seeds for data sampling. Raw JSON results are archived at results/cloud_benchmark/ in the Solstice-EIM repository. The benchmark script supports any HuggingFace causal LM and can be extended to additional models as access becomes available.