# What Is LLM Tokenization? Learn with Interactive Visualizations

Canonical: https://llm-explained.vicoworks.com/en/learn/tokenization/
Language: en
Publisher: Vicoworks (https://vicoworks.com)
Updated: 2026-10-25

A model sees token IDs. Tokens can be words, word pieces, punctuation, or characters including a leading space.

The tokenizer maps text pieces to vocabulary IDs. A token is not always a whole word.

## How it works
### Input text
Text is the starting point. This profile uses a toy vocabulary rather than the GPT-2 tokenizer.

### Split tokens
Leading spaces belong to tokens. Presets retain pieces and offsets that reconstruct the input.

### Lookup ID
Positions 0, 1, 2, … differ from vocabulary IDs. An ID does not measure the strength of meaning.

## What can you observe in the lab?
Separate capsules show token boundaries. Colors follow token positions through the journey.

## Reference scale and simulation limits
Visuals use illustrative flows and fixtures. GPT-3 175B has 96 blocks, 96 heads per block, width 12,288, and a 2,048-token context. Its paper alternates dense and locally banded sparse attention; this lab examines causal dense attention. The numerical workbench uses a toy model with two blocks, two heads, width 8, and a vocabulary of 16. GPT-3 weights are not loaded, and presets do not generate answers from a trained language model.

## Primary sources
- [GPT-3 · Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
- [GPT-2 · reference implementation](https://github.com/openai/gpt-2/blob/master/src/model.py)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
- [Word representations · Mikolov et al.](https://arxiv.org/abs/1301.3781)
- [KV cache · Hugging Face](https://huggingface.co/docs/transformers/main/cache_explanation)