Vector Analogies and Static Embeddings

How it works

Vectors illustrate relationships between representations. Tokyo − Japan + France is a static-embedding example, separate from Transformer inference.

1. Source vectors

   These are four-dimensional static toy word vectors, separate from Transformer inference.

2. Vector arithmetic

   Arithmetic operates on source vectors. The lab displays a two-dimensional projection.

3. Ranking cosine

   The result approaches the selected target in this fixture. It is not identical or universal.

What can you observe in the lab?

The axes are illustrative projections. Real models have thousands of features; contextual representations change with context.

The Scanner opens notes. The Token tracer selects token positions. The Head lens compares two demo heads. The Time controller pauses signals. The Calibrator changes experiment conditions before you predict and test the outcome.

What to remember

Visuals use illustrative flows and fixtures. GPT-3 175B has 96 blocks, 96 heads per block, width 12,288, and a 2,048-token context. Its paper alternates dense and locally banded sparse attention; this lab examines causal dense attention. The numerical workbench uses a toy model with two blocks, two heads, width 8, and a vocabulary of 16. GPT-3 weights are not loaded, and presets do not generate answers from a trained language model.

Primary sources

• GPT-3 · Language Models are Few-Shot Learners

  175B scale: 96 blocks, width 12,288, 96 heads, context 2,048; alternating dense and locally banded sparse attention.

• GPT-2 · reference implementation

  Pre-norm, learned positions, GELU, residuals, and tied embeddings.

• Attention Is All You Need

  Foundations of scaled dot-product and multi-head attention.

• Word representations · Mikolov et al.

  Background on static word embeddings and geometry.

• KV cache · Hugging Face

  Prefill, decoding, and K/V reuse.