# Query, Key, Value: How LLM Attention Works

Canonical: https://llm-explained.vicoworks.com/en/learn/query-key-value/
Language: en
Publisher: Vicoworks (https://vicoworks.com)
Updated: 2026-10-25

Queries match patterns, Keys offer patterns to match, and Values carry information to mix. All three are learned projections of the same representation.

Within each block, LayerNorm normalizes features per token. Three learned projections produce Query, Key, and Value, then split features into heads. These stations examine parts of the same block.

## How it works
### Pre-normalization
Compute the mean and variance across each token's eight features. The demo uses gamma=1 and beta=0.

### Query projection
Click an output to inspect eight multiplication pairs between an input row and a weight column.

### Key projection
Keys form patterns matched by Queries. Weights stay fixed; activations depend on input.

### Value projection
Values carry the information mixed once attention weights are available.

### Split two heads
Width 8 is split into 2 heads × 4 features. Changing heads changes Keys, Values, scores, and outputs.

## What can you observe in the lab?
One stream branches three ways: Queries look for matches, Keys are matched, and Values carry information.

## Reference scale and simulation limits
Visuals use illustrative flows and fixtures. GPT-3 175B has 96 blocks, 96 heads per block, width 12,288, and a 2,048-token context. Its paper alternates dense and locally banded sparse attention; this lab examines causal dense attention. The numerical workbench uses a toy model with two blocks, two heads, width 8, and a vocabulary of 16. GPT-3 weights are not loaded, and presets do not generate answers from a trained language model.

## Primary sources
- [GPT-3 · Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
- [GPT-2 · reference implementation](https://github.com/openai/gpt-2/blob/master/src/model.py)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
- [Word representations · Mikolov et al.](https://arxiv.org/abs/1301.3781)
- [KV cache · Hugging Face](https://huggingface.co/docs/transformers/main/cache_explanation)