# Vector Analogies and Static Embeddings

Canonical: https://llm-explained.vicoworks.com/en/learn/vector-analogies/
Language: en
Publisher: Vicoworks (https://vicoworks.com)
Updated: 2026-10-25

Vector arithmetic illustrates relationships in static word embeddings. This example is not the literal mechanism by which GPT answers, and the relationship is not universal.

Vectors illustrate relationships between representations. Tokyo − Japan + France is a static-embedding example, separate from Transformer inference.

## How it works
### Source vectors
These are four-dimensional static toy word vectors, separate from Transformer inference.

### Vector arithmetic
Arithmetic operates on source vectors. The lab displays a two-dimensional projection.

### Ranking cosine
The result approaches the selected target in this fixture. It is not identical or universal.

## What can you observe in the lab?
The axes are illustrative projections. Real models have thousands of features; contextual representations change with context.

## Reference scale and simulation limits
Visuals use illustrative flows and fixtures. GPT-3 175B has 96 blocks, 96 heads per block, width 12,288, and a 2,048-token context. Its paper alternates dense and locally banded sparse attention; this lab examines causal dense attention. The numerical workbench uses a toy model with two blocks, two heads, width 8, and a vocabulary of 16. GPT-3 weights are not loaded, and presets do not generate answers from a trained language model.

## Primary sources
- [GPT-3 · Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
- [GPT-2 · reference implementation](https://github.com/openai/gpt-2/blob/master/src/model.py)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
- [Word representations · Mikolov et al.](https://arxiv.org/abs/1301.3781)
- [KV cache · Hugging Face](https://huggingface.co/docs/transformers/main/cache_explanation)