AMD Benchmarks Five Speculative Decoding Methods in vLLM — SkimNews
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- vLLM's speculative decoding uses a draft-and-verify loop where a lightweight draft component proposes candidate tokens and the target model verifies them in a single pass, committing multiple output tokens per round while preserving the target model's output behavior.
- AMD's team tested five drafting methods on AMD Instinct MI300X and MI355X GPUs running the ROCm platform: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark — each differing in how it receives target-model information and whether it generates candidates sequentially, in parallel, or via a hybrid approach.
- Native MTP modules are built directly into the target-model architecture and generate candidate tokens sequentially, while separate MTP drafters (like Gemma 4 MTP) use a distinct checkpoint paired with a specific target model and share the target's KV cache during inference.
- Dedicated target-conditioned draft networks split further: EAGLE-3 drafts autoregressively from target-model hidden states, DFlash drafts parallel blocks from the same hidden states, and DSpark adds lightweight causal correction with confidence-based prefix selection.
- Throughput results varied significantly across drafting methods and proposal lengths, and also depended on the model family, draft checkpoint, workload, and acceptance behavior — with the source noting no single method dominated across all conditions.
- vLLM reuses the MTP path through additional forward passes when the configured speculative length exceeds the checkpoint's direct prediction depth, allowing longer proposals at the cost of more sequential drafting work.
Why it matters: AMD's benchmarks give ROCm-based deployments concrete data for choosing among speculative decoding strategies on MI300X and MI355X hardware — a practical decision since the source shows throughput gains are method-specific rather than universal, meaning serving teams must match the draft architecture to their model and workload rather than expecting a default win.
Ask SkimNews
