A-TIER technique
Claude — Attention is a dot product between a query vector and a key vector. A plain dot...
100k ft — What is this?
Claude — Attention is a dot product between a query vector and a key vector. A plain dot...
50k ft — Why care?
Same tokens. Different order. Different answer. Something inside the model is tracking position — and it’s not what m... — @adam_rosler; uses Claude, DeepSeek, Gemini.
10k ft — Key takeaways
(no takeaways yet)
5k ft — Tools & techniques
Claude, DeepSeek, Gemini, Llama
(no techniques tagged)
1k ft — Deep-dive / try it
Same tokens. Different order. Different answer. Something inside the model is tracking position — and it’s not what most explainers tell you.
Attention is a dot product between a query vector and a key vector. A plain dot product doesn’t know which token came first. The fix is Rotary Position Embedding (RoPE): before the dot product, both vectors get rotated by an angle proportional to their position. Token one, a small rotation. Token fifty, a much bigger one.
Now the dot product depends on the difference between two positions. That difference IS the model’s sense of order. Llama, GPT, Gemini, Claude, DeepSeek — every attention head in every layer rotates vectors before comparing them.
Position is encoded as rotation, not addition. That’s how your model knows what comes before what.
yo…
Transcript excerpt
You shuffled the words in your prompt and got a completely different answer. Same tokens. Different order. Different answer. Something inside the model is tracking position. Attention works like this. Every token builds two vectors, a query, and a key. The model compares them with a dot product. Big number means these two tokens are related. But a dot product has no idea which token came first. So before the comparison, both vectors get rotated by an angle proportional to their position. Token one, small rotation. Token 50, bigger rotation. Now when query and key meet, their dot product depends on the difference between their positions. That difference is how the model feels order. Position is encoded as rotation, not addition. This is called rotary position embedding. Lama uses it. GPT uses it. Gemini, Claude, deepseek, all of them. Every attention head, every layer is rotating vectors before the dot product. That is how your model knows what comes before what. Query, key, rotate.