Local LLM Output Diverges From Labs Across Backends, Quant — SkimNews
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- The author ran Qwen3.6-27B (BF16) on an RTX PRO 6000 Blackwell GPU inside a pinned nightly vLLM build containing 734 Python packages, replaying a ~100k-token "Prompt 2" captured from a real Turnstone lab workstream to measure implementation-specific inference divergence.
- Test 1 compared FlashAttention 2, Flash Inference, and Triton Attention backends; greedy next-token choices matched across the first several thousand tokens but diverged in clusters later in the prompt, with the author noting disagreements tracked prompt content rather than context length.
- Same-backend cross-GPU repeatability runs produced bit-for-bit identical logits at every hidden state, meaning the attention-backend divergence comes exclusively from the matrix-multiplication-and-addition operations during prefill — not from weights or sampling.
- Test 2 left BF16 weights and activations untouched while quantizing the KV cache: BF16 was fine, INT8 eventually recovered from tool-call token flips, and INT4 did not recover at all — producing a reproducible tool-calling error.
- The author warns that impossibly low KL-divergence claims on quantized Hugging Face model cards are uninterpretable without full disclosure of reference checkpoints, runtime environment, evaluation text, calibration data, context lengths, sampled positions, KL direction, vocabulary truncation, and aggregation methods.
- Sampler settings on the model card (e.g., temp 1.0, top-p 0.95) are model-specific; the author notes setting temperature too low is why Qwen's THINK output loops indefinitely and fails to escape.
- Test 3 adds BF16, official FP8, INT8 W8A16, NVIDIA NVFP4, and AWQ W4A16 weight variants of Qwen3.6-27B to compare across GEMM kernel paths and precision formats.
Why it matters: The author shows local LLM users are not running the "same model" as the lab — every inference engine routes through a distinct stack of attention backends, CUDA kernels, and quantizations that pick different next tokens from identical weights, and the proof is that BF16 can tool-call where INT4 KV-cache cannot. Popular quantized-model-card KLD numbers are unverifiable without methodology disclosure, so benchmark shopping is meaningless absent that transparency.
Ask SkimNews
