NEUROLAB / Created by VICOWORKS · 2026-10-25
Transformer Blocks: Residuals, LayerNorm, MLP, and GELU
Each block contains attention and an MLP. Two residual connections retain input paths. LayerNorm operates across each token's feature dimension.
Try it in the 3D lab →How it works
Attention output is added to the residual stream. A second LayerNorm and an MLP with GELU process features per token, followed by a second residual addition. The block repeats 96 times with different weights.
Residual attention
The residual adds the initial representation to the attention output, retaining the input.
Pre-norm MLP
The second LayerNorm operates on residual output across features per token.
Expand 8 → 32
The MLP expands the feature dimension fourfold, independently per token.
GELU activation
GELU is a smooth curve; negative values are not all set to zero.
Project 32 → 8
The second projection restores the dimension for matching residual shapes.
Residual MLP
Every block has two residuals and both sublayers. GPT-3 reference: 96 blocks; fixture: 2 blocks.
What can you observe in the lab?
The outer paths bypass attention and MLP: these are residual connections. The 96 plates represent model depth rather than decoding steps.
The Scanner opens notes. The Token tracer selects token positions. The Head lens compares two demo heads. The Time controller pauses signals. The Calibrator changes experiment conditions before you predict and test the outcome.
What to remember
Visuals use illustrative flows and fixtures. GPT-3 175B has 96 blocks, 96 heads per block, width 12,288, and a 2,048-token context. Its paper alternates dense and locally banded sparse attention; this lab examines causal dense attention. The numerical workbench uses a toy model with two blocks, two heads, width 8, and a vocabulary of 16. GPT-3 weights are not loaded, and presets do not generate answers from a trained language model.
Primary sources
- GPT-3 · Language Models are Few-Shot Learners
175B scale: 96 blocks, width 12,288, 96 heads, context 2,048; alternating dense and locally banded sparse attention.
- GPT-2 · reference implementation
Pre-norm, learned positions, GELU, residuals, and tied embeddings.
- Attention Is All You Need
Foundations of scaled dot-product and multi-head attention.
- Word representations · Mikolov et al.
Background on static word embeddings and geometry.
- KV cache · Hugging Face
Prefill, decoding, and K/V reuse.