Local LLM divergence traced to inference stack
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Qwen3.6-27B tested on an RTX PRO 6000 Blackwell GPU across three vLLM attention backends (FlashAttention 2, Flash Inference, Triton) showed top-1 token agreement for the first several thousand tokens of a ~100k-token workstream, then began disagreeing on the next-token choice in later portions of the prompt.
- Same-backend repeatability runs were "bit for bit identical" at every hidden state, proving the inter-backend divergence comes exclusively from matrix-multiplication operations during prefill inside the attention kernels—not random noise.
- KV-cache quantization at int4 produced reproducible tool-calling errors that never recovered, while BF16 ran cleanly and int8 eventually recovered, using a real workstream captured from a Turnstone lab containing multiple tool calls.
- Token disagreements between backends appeared in clusters tied to prompt content rather than scaling smoothly with context length, undercutting any notion of a single context threshold at which the model "falls apart."
- The author warns that impossibly low KL-divergence claims on quantized model cards are uninterpretable without full disclosure of reference checkpoints, runtime environment, evaluation text, calibration data, context lengths, sampled positions, KL direction, vocabulary truncation, and aggregation methodology.
- Weight-quantization formats compared in the study include the BF16 reference, official FP8, INT8 W8A16, Nvidia's NVFP4, and AWQ W4A16 builds of Qwen3.6-27B, with a pinned nightly vLLM container exposing 734 packages (252 Python) along the inference path.
Why it matters: For practitioners running local LLMs, the study reframes poor observed performance as an inference-stack problem rather than a model-quality problem: attention-backend choice and KV-cache precision directly determine whether a 100k-token agentic workload completes tool calls or silently breaks. The irrecoverable int4 KV-cache tool-calling failure specifically makes quant selection a reliability decision, not just a memory-saving one.
Ask SkimNews
