What Is LLM Tokenization? Learn with Interactive Visualizations

How it works

The tokenizer maps text pieces to vocabulary IDs. A token is not always a whole word.

1. Input text

   Text is the starting point. This profile uses a toy vocabulary rather than the GPT-2 tokenizer.

2. Split tokens

   Leading spaces belong to tokens. Presets retain pieces and offsets that reconstruct the input.

3. Lookup ID

   Positions 0, 1, 2, … differ from vocabulary IDs. An ID does not measure the strength of meaning.

What can you observe in the lab?

Separate capsules show token boundaries. Colors follow token positions through the journey.

The Scanner opens notes. The Token tracer selects token positions. The Head lens compares two demo heads. The Time controller pauses signals. The Calibrator changes experiment conditions before you predict and test the outcome.

What to remember

Visuals use illustrative flows and fixtures. GPT-3 175B has 96 blocks, 96 heads per block, width 12,288, and a 2,048-token context. Its paper alternates dense and locally banded sparse attention; this lab examines causal dense attention. The numerical workbench uses a toy model with two blocks, two heads, width 8, and a vocabulary of 16. GPT-3 weights are not loaded, and presets do not generate answers from a trained language model.

Primary sources

• GPT-3 · Language Models are Few-Shot Learners

  175B scale: 96 blocks, width 12,288, 96 heads, context 2,048; alternating dense and locally banded sparse attention.

• GPT-2 · reference implementation

  Pre-norm, learned positions, GELU, residuals, and tied embeddings.

• Attention Is All You Need

  Foundations of scaled dot-product and multi-head attention.

• Word representations · Mikolov et al.

  Background on static word embeddings and geometry.

• KV cache · Hugging Face

  Prefill, decoding, and K/V reuse.