Speculative Decoding in vLLM on AMD GPUs — SkimNews
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- vLLM explores speculative decoding, a draft-and-verify mechanism where a lightweight draft component proposes candidate tokens that the target model verifies in a single pass, allowing several output tokens to be committed from one verification step when accepted.
- Five drafting methods are examined: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark, which fall into three architectural categories — native MTP modules built into the target model, separate MTP drafters, and dedicated target-conditioned draft networks.
- EAGLE-3, DFlash, and DSpark are separate speculator models trained for specific target models: EAGLE-3 drafts autoregressively from target hidden states, DFlash drafts parallel blocks from hidden states, and DSpark adds lightweight causal correction with confidence-based prefix selection.
- Native MTP uses a model-native auxiliary prediction path that combines a hidden representation from the target model with a token embedding, generating candidate tokens sequentially before a single target-model verification pass.
- Gemma 4 MTP is a separately packaged draft component with its own checkpoint that shares the target model's KV cache and uses target-model activations during inference, keeping it closely connected to the target model.
- Experiments were conducted on AMD Instinct MI300X and MI355X GPUs using the ROCm open software platform, with the authors noting that throughput impact varied across drafting methods, proposal lengths, model families, draft checkpoints, workloads, and acceptance behavior.
- vLLM can reuse the MTP path through additional forward passes when
num_speculative_tokensexceeds the checkpoint's prediction depth, proposing more candidates per round at the cost of more sequential drafting work.
Why it matters: For teams serving LLMs on AMD's ROCm platform, the post provides concrete evidence that speculative decoding gains are not uniform — throughput swings depend on which drafting method pairs with which model family, meaning practitioners must benchmark per workload rather than assume a single speedup factor. AMD gains a validated playbook for MI300X and MI355X deployments, narrowing the optimization gap that historically favored NVIDIA-only serving stacks.
Ask SkimNews
