NEUROLAB / Created by VICOWORKS · 2026-10-25
Query, Key, Value: How LLM Attention Works
Queries match patterns, Keys offer patterns to match, and Values carry information to mix. All three are learned projections of the same representation.
Try it in the 3D lab →How it works
Within each block, LayerNorm normalizes features per token. Three learned projections produce Query, Key, and Value, then split features into heads. These stations examine parts of the same block.
Pre-normalization
Compute the mean and variance across each token's eight features. The demo uses gamma=1 and beta=0.
Query projection
Click an output to inspect eight multiplication pairs between an input row and a weight column.
Key projection
Keys form patterns matched by Queries. Weights stay fixed; activations depend on input.
Value projection
Values carry the information mixed once attention weights are available.
Split two heads
Width 8 is split into 2 heads × 4 features. Changing heads changes Keys, Values, scores, and outputs.
What can you observe in the lab?
One stream branches three ways: Queries look for matches, Keys are matched, and Values carry information.
The Scanner opens notes. The Token tracer selects token positions. The Head lens compares two demo heads. The Time controller pauses signals. The Calibrator changes experiment conditions before you predict and test the outcome.
What to remember
Visuals use illustrative flows and fixtures. GPT-3 175B has 96 blocks, 96 heads per block, width 12,288, and a 2,048-token context. Its paper alternates dense and locally banded sparse attention; this lab examines causal dense attention. The numerical workbench uses a toy model with two blocks, two heads, width 8, and a vocabulary of 16. GPT-3 weights are not loaded, and presets do not generate answers from a trained language model.
Primary sources
- GPT-3 · Language Models are Few-Shot Learners
175B scale: 96 blocks, width 12,288, 96 heads, context 2,048; alternating dense and locally banded sparse attention.
- GPT-2 · reference implementation
Pre-norm, learned positions, GELU, residuals, and tied embeddings.
- Attention Is All You Need
Foundations of scaled dot-product and multi-head attention.
- Word representations · Mikolov et al.
Background on static word embeddings and geometry.
- KV cache · Hugging Face
Prefill, decoding, and K/V reuse.