# Transformer Blocks: Residuals, LayerNorm, MLP, and GELU

Canonical: https://llm-explained.vicoworks.com/en/learn/transformer-block/
Language: en
Publisher: Vicoworks (https://vicoworks.com)
Updated: 2026-10-25

Each block contains attention and an MLP. Two residual connections retain input paths. LayerNorm operates across each token's feature dimension.

Attention output is added to the residual stream. A second LayerNorm and an MLP with GELU process features per token, followed by a second residual addition. The block repeats 96 times with different weights.

## How it works
### Residual attention
The residual adds the initial representation to the attention output, retaining the input.

### Pre-norm MLP
The second LayerNorm operates on residual output across features per token.

### Expand 8 → 32
The MLP expands the feature dimension fourfold, independently per token.

### GELU activation
GELU is a smooth curve; negative values are not all set to zero.

### Project 32 → 8
The second projection restores the dimension for matching residual shapes.

### Residual MLP
Every block has two residuals and both sublayers. GPT-3 reference: 96 blocks; fixture: 2 blocks.

## What can you observe in the lab?
The outer paths bypass attention and MLP: these are residual connections. The 96 plates represent model depth rather than decoding steps.

## Reference scale and simulation limits
Visuals use illustrative flows and fixtures. GPT-3 175B has 96 blocks, 96 heads per block, width 12,288, and a 2,048-token context. Its paper alternates dense and locally banded sparse attention; this lab examines causal dense attention. The numerical workbench uses a toy model with two blocks, two heads, width 8, and a vocabulary of 16. GPT-3 weights are not loaded, and presets do not generate answers from a trained language model.

## Primary sources
- [GPT-3 · Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
- [GPT-2 · reference implementation](https://github.com/openai/gpt-2/blob/master/src/model.py)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
- [Word representations · Mikolov et al.](https://arxiv.org/abs/1301.3781)
- [KV cache · Hugging Face](https://huggingface.co/docs/transformers/main/cache_explanation)