Context Window Optimization: KV Cache Eviction and Hierarchical Attention

The expansion of context windows from 8k tokens to 1 million+ tokens has eliminated many traditional chunking constraints. However, in production serving infrastructure, native full-attention scaling remains prohibitively expensive: the memory footprint of the Key-Value (KV) cache grows linearly with context length $L$:

$$\text{KV Cache Size} = 2 \times B \times L \times N_{\text{layers}} \times N_{\text{kv\_heads}} \times d_{\text{head}} \times \text{BytesPerElement}$$

For a 70B model processing 128k context with batch size 8, the raw KV cache alone exceeds $180 \text{ GB}$ of GPU memory, bottlenecking throughput and pricing out high-concurrency workflows.

1. Dynamic KV Cache Eviction Topologies

To compress memory footprint without degrading recall, modern inference runtimes implement dynamic token retention policies:

flowchart TD
    Stream["Incoming Token Sequence"] --> AttnMap["Attention Score Analysis"]
    AttnMap --> Sink["Attention Sinks (Initial 4-8 System Tokens) [Always Retained]"]
    AttnMap --> Local["Sliding Window Buffer (Latest 1024 Tokens) [High Recency]"]
    AttnMap --> Heavy["Heavy Hitters (Top Cumulative Attention Tokens) [Sparse Retention]"]
    AttnMap --> Evict["Transient Punctuation & Fillers [Evicted from Cache]"]
    Sink --> ActiveKV["Compressed Active KV Cache (70% Footprint Reduction)"]
    Local --> ActiveKV
    Heavy --> ActiveKV

2. Attention Sinks and StreamingLLM

StreamingLLM demonstrated a foundational insight: the initial tokens in a sequence absorb disproportionately massive attention scores regardless of semantic relevance (termed Attention Sinks).

If the initial 4 tokens are evicted from the KV cache, model perplexity instantly explodes to infinity. By permanently pinning the first 4 tokens in memory alongside a rolling local window, models can stream over infinite context horizons without catastrophic attention collapse.

3. Quantitative Comparison of Compression Architectures

4. Architectural Rules for Agent Systems

• Leverage Prefix Caching: Ensure system instructions and MCP tool schemas appear at the very beginning of prompts so that their computed KV caches can be shared identically across all concurrent requests.
• Apply Hierarchical RAG for Long Repositories: Do not indiscriminately dump 50 files into context. Use lightweight BM25 + dense vector pre-filtering to bound active context within optimal attention ranges.
• Monitor Attention Dispersion: Track entropy of attention weights; high entropy indicates the model is drowning in context noise, signaling the need for intermediate summarization.