A transformer reads a word by letting it ask a question. Each word turns into a query; every word also offers a key. The head scores the query against every key with a dot product, softmaxes the scores, and mixes the words in those proportions. That mix is all “attention” is.
Here the keys are plotted as points and you hold the query in your hand. Aim it toward a word’s key and that thread thickens; aim far and the softmax sharpens until one word wins — the head has focused.
It is not measuring how close you are. The dashed spur drops a key onto your aim line, and that shadow — times how far out you have aimed — is the score. So park the query right on top of a short function word and a long key that lines up with your aim can still take the attention. Dot product, not distance.
The hollow marker is the fourth operation: the mix, o = Σ wi vi — and in this toy each word’s value is its key, so the mix lands in the same space. Focus the head and it slides out toward the winning word: the head presets carry it about three-fifths of the way there (|o| ≈ 0.63 against a key at 1.06), and a hard drag into a corner parks it almost on the word itself (0.97 of 1.02). Flatten the gaze and it collapses onto the crosshair, because the average of all nine words is almost nothing. That collapse is what high entropy costs you.
Aim at nothing — the middle — and every score is zero, so attention spreads evenly. Every score here is divided by √d (d = 2 in this toy, so √d ≈ 1.41). The reason is the knob already in your hand: sharpness rides on the size of the dot products, so without that divisor scores would grow with model width and the softmax would saturate into a hard argmax.
What makes a head a head is the pair of projections — how the query map is turned relative to the key map. Leave the query on its own word and turn the key projection: at −112° the verb reads its subject, at +123° the same head reads its object. Nothing else changed. That angle is what training finds.
Something in the simulation stopped unexpectedly — the lesson continues without it. Nothing you did was wrong; you can move on.