What Is a KV Cache? Understanding LLM Prefill and Decoding

How it works

Prefill processes the prompt. During decoding, previous Keys and Values are retained per layer. The new Query reads the cache. The selected token is appended and the process repeats until a limit or end token.

1. Store prefix K/V

   Prefill processes the prompt with a causal mask and stores K/V activations per layer.

2. Decode a new token

   The new Query sees all available Keys. Prefix K/V are not projected again.

3. Repeat autoregression

   The new token joins the next input. Prefix edits invalidate the dependent cache.

What can you observe in the lab?

Racks retain K and V streams. The return path carries the new token into the model. The cache does not store prepared answers.

The Scanner opens notes. The Token tracer selects token positions. The Head lens compares two demo heads. The Time controller pauses signals. The Calibrator changes experiment conditions before you predict and test the outcome.

What to remember

Visuals use illustrative flows and fixtures. GPT-3 175B has 96 blocks, 96 heads per block, width 12,288, and a 2,048-token context. Its paper alternates dense and locally banded sparse attention; this lab examines causal dense attention. The numerical workbench uses a toy model with two blocks, two heads, width 8, and a vocabulary of 16. GPT-3 weights are not loaded, and presets do not generate answers from a trained language model.

Primary sources

• GPT-3 · Language Models are Few-Shot Learners

  175B scale: 96 blocks, width 12,288, 96 heads, context 2,048; alternating dense and locally banded sparse attention.

• GPT-2 · reference implementation

  Pre-norm, learned positions, GELU, residuals, and tied embeddings.

• Attention Is All You Need

  Foundations of scaled dot-product and multi-head attention.

• Word representations · Mikolov et al.

  Background on static word embeddings and geometry.

• KV cache · Hugging Face

  Prefill, decoding, and K/V reuse.