# Transformers Intuition

**Track:** Model Foundations — AI for Entrepreneurship — model foundations (5)
**Framework / surface:** venture strategy
**Level:** Intermediate
**Prerequisites:** How Models Learn, Noise → Structure: Diffusion Intuition

**In one line:** Attention weighs every token against every other; the context window is the model's entire working memory, and whatever is not in it does not exist.

## Theory, aesthetics & inspiration

The transformer, introduced in "Attention Is All You Need" (Vaswani et al., 2017), replaced reading in order with attention: text is split into tokens, and every token computes how much every token before it matters, in parallel, layer after layer. That is the whole trick — relationships, not sequence, weighted and re-weighted until the next token falls out. The context window is the span attention can see, and because attention compares tokens against each other, cost climbs steeply as the window grows — which is why long context is metered and why models forget what falls outside it. The practical intuition for creative work: the window is the model's entire working memory. Whatever is not in it does not exist. Prompts, references, excerpts, and examples are not commands so much as material — what you place in the window is what the attention has to work with.

**Founder question:** What would you place in the window if you treated it as a moodboard rather than a command line?