A-TIER technique
Llama — Engineers have been quantizing the cache down for years. 8 bits works. 4 bits...
100k ft — What is this?
Llama — Engineers have been quantizing the cache down for years. 8 bits works. 4 bits...
50k ft — Why care?
Every token you send to an LLM leaves a trail. The model stores a key and a value vector per token, in full precision... — @adam_rosler; uses Llama, Stable Diffusion.
10k ft — Key takeaways
(no takeaways yet)
5k ft — Tools & techniques
Llama, Stable Diffusion
(no techniques tagged)
1k ft — Deep-dive / try it
Every token you send to an LLM leaves a trail. The model stores a key and a value vector per token, in full precision. That’s the KV cache. At 32k tokens on Llama 70B, it’s 17 gigabytes of GPU memory per request, and it only grows.
Engineers have been quantizing the cache down for years. 8 bits works. 4 bits mostly works. 3 bits always broke accuracy — until Google’s TurboQuant, accepted at ICLR 2026.
The trick: rewrite each vector in polar form. The angle distribution is known, so you only keep the radius and a few bits of angle. That’s PolarQuant. A Johnson-Lindenstrauss random rotation collapses the rest to sign bits.
Result: 3 bits per number instead of 16. Roughly 5x less memory. No finetuning, no calibration data, drops into any transformer. Long context for the price of short con…
Transcript excerpt
Every token you send leaves a trail. That trail doesn't shrink. It grows five times bigger. Every prompt saves two vectors per token, a key, a value. That's the KV cash. At 32,000 tokens on Lama 70B, it's 17 gigabytes per request. Every new token makes it bigger, so engineers quantize, squeeze the numbers down, but three bit always broke accuracy. Google's trick rewrite each vector in polar form, radius and angle. The angles follow a known pattern so you only keep a few bits. That's polar quant. A Johnson-Lindon Strauss rotation handles the rest with sign bits. Three bits per number instead of 16, five times less memory, no fine tuning, no calibration, drops into any transformer. Compress the cash, keep the accuracy, ship longer context.