Token Embeddings and Positions in LLMs

How it works

Each ID retrieves an embedding vector. Adding a position embedding gives the same token different input representations at different positions.

1. Lookup embedding

   An ID selects a learned embedding row; this fixture has 16 rows.

2. Position lookup

   GPT-2 uses learned position embeddings. Positions here also use lookups rather than sinusoidal encoding.

3. Add position

   Addition is elementwise. Swapping order preserves token embeddings but changes position contributions.

What can you observe in the lab?

Light columns represent vector features. Colors illustrate values rather than named semantic axes.

The Scanner opens notes. The Token tracer selects token positions. The Head lens compares two demo heads. The Time controller pauses signals. The Calibrator changes experiment conditions before you predict and test the outcome.

What to remember

Visuals use illustrative flows and fixtures. GPT-3 175B has 96 blocks, 96 heads per block, width 12,288, and a 2,048-token context. Its paper alternates dense and locally banded sparse attention; this lab examines causal dense attention. The numerical workbench uses a toy model with two blocks, two heads, width 8, and a vocabulary of 16. GPT-3 weights are not loaded, and presets do not generate answers from a trained language model.

Primary sources

• GPT-3 · Language Models are Few-Shot Learners

  175B scale: 96 blocks, width 12,288, 96 heads, context 2,048; alternating dense and locally banded sparse attention.

• GPT-2 · reference implementation

  Pre-norm, learned positions, GELU, residuals, and tied embeddings.

• Attention Is All You Need

  Foundations of scaled dot-product and multi-head attention.

• Word representations · Mikolov et al.

  Background on static word embeddings and geometry.

• KV cache · Hugging Face

  Prefill, decoding, and K/V reuse.