# Causal Attention and Masking in Transformers

Canonical: https://llm-explained.vicoworks.com/en/learn/causal-attention/
Language: en
Publisher: Vicoworks (https://vicoworks.com)
Updated: 2026-10-25

Query–Key scores are divided by the square root of head dimension. Future positions are masked, then softmax produces weights to mix Values.

QKᵀ produces token-pair scores. Scores are divided by √128, future positions are masked, and softmax produces weights to mix Values.

## How it works
### Multiply Q × K
Each cell compares one Query row with one Key row. Click to inspect all multiplications.

### Sum the dot product
Four products are summed into one score. Scores are not yet probabilities.

### Scale
Dividing by √d_head controls score scale. Here, √4 = 2.

### Causal mask
Future positions are unavailable. A score of −∞ becomes exactly zero weight after softmax.

### Row-wise softmax
Normalize across Key columns rather than the entire matrix. Every row sums to 1.

### Mix Values
Attention weights and Value contributions are different. Click an output to inspect its weighted sum.

### Concatenate heads
Two width-4 outputs are concatenated back to width 8.

### Output projection
A learned output projection mixes features across heads before residual addition.

## What can you observe in the lab?
Lines point only to current and preceding positions. Thickness illustrates weights; other heads may have different patterns.

## Reference scale and simulation limits
Visuals use illustrative flows and fixtures. GPT-3 175B has 96 blocks, 96 heads per block, width 12,288, and a 2,048-token context. Its paper alternates dense and locally banded sparse attention; this lab examines causal dense attention. The numerical workbench uses a toy model with two blocks, two heads, width 8, and a vocabulary of 16. GPT-3 weights are not loaded, and presets do not generate answers from a trained language model.

## Primary sources
- [GPT-3 · Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
- [GPT-2 · reference implementation](https://github.com/openai/gpt-2/blob/master/src/model.py)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
- [Word representations · Mikolov et al.](https://arxiv.org/abs/1301.3781)
- [KV cache · Hugging Face](https://huggingface.co/docs/transformers/main/cache_explanation)