Google's TurboQuant Cuts LLM Memory 6x, Speeds Up 8x
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Google Research unveiled TurboQuant, a compression algorithm that cuts LLM key-value cache memory usage by up to 6x and increases performance 8x in some tests with no quality loss.
- The two-step process starts with PolarQuant, which converts vector coordinates from XYZ Cartesian to polar form (radius + direction), creating a compact "shorthand" for storage and skipping expensive normalization steps.
- A second step called Quantized Johnson-Lindenstrauss (QJL) applies a 1-bit error-correction layer (+1 or -1 per vector) to smooth residual errors and preserve the relationship data that attention scoring depends on.
- Google tested TurboQuant on Gemma and Mistral open models, achieving perfect downstream results across a suite of long-context benchmarks while compressing the cache to just 3 bits with no additional training required, making it drop-in compatible with existing models.
- On Nvidia H100 accelerators, computing attention scores with 4-bit TurboQuant ran 8x faster than with 32-bit unquantized keys.
- Google flags mobile AI as the likeliest near-term beneficiary, since smartphone hardware limits currently force on-device models to be small — TurboQuant could boost output quality without sending user data to the cloud.
Why it matters: TurboQuant gives AI labs and cloud providers a concrete choice per Google's own framing: slash the memory costs of running current models, or pour the freed capacity into larger, more complex ones. The company explicitly flags mobile devices as the biggest winner, since on-device AI gains quality and speed without a cloud round-trip — a direct consequence of the 6x memory and 8x speed gains Google reports in its benchmarks.
