# Token Embeddings and Positions in LLMs

Canonical: https://llm-explained.vicoworks.com/en/learn/embeddings-position/
Language: en
Publisher: Vicoworks (https://vicoworks.com)
Updated: 2026-10-25

An ID selects an embedding row. Adding a learned position vector encodes order. The ID itself is not a numerical measure of meaning.

Each ID retrieves an embedding vector. Adding a position embedding gives the same token different input representations at different positions.

## How it works
### Lookup embedding
An ID selects a learned embedding row; this fixture has 16 rows.

### Position lookup
GPT-2 uses learned position embeddings. Positions here also use lookups rather than sinusoidal encoding.

### Add position
Addition is elementwise. Swapping order preserves token embeddings but changes position contributions.

## What can you observe in the lab?
Light columns represent vector features. Colors illustrate values rather than named semantic axes.

## Reference scale and simulation limits
Visuals use illustrative flows and fixtures. GPT-3 175B has 96 blocks, 96 heads per block, width 12,288, and a 2,048-token context. Its paper alternates dense and locally banded sparse attention; this lab examines causal dense attention. The numerical workbench uses a toy model with two blocks, two heads, width 8, and a vocabulary of 16. GPT-3 weights are not loaded, and presets do not generate answers from a trained language model.

## Primary sources
- [GPT-3 · Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
- [GPT-2 · reference implementation](https://github.com/openai/gpt-2/blob/master/src/model.py)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
- [Word representations · Mikolov et al.](https://arxiv.org/abs/1301.3781)
- [KV cache · Hugging Face](https://huggingface.co/docs/transformers/main/cache_explanation)