DeepSeek-V4 Latent Reasoning Ships as Standalone Model
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- DeepSeek-V4-Flash-0731-Latent-Reasoning ships as a complete model rather than a grafted-on adapter: backbone quantized to NVFP4 (group size 16) on routed MoE experts, plus the DSpark draft block and a 35.7M-param latent reasoning head, all in one HuggingFace repo weighing roughly 79 GiB per GPU at TP=2.
- The CoLaR-style head reads the backbone's layer-35 4096-d hidden state, projects it through a variational compression ([mu, log_sigma], log_sigma clamped) into a 1024-d latent, then decodes it back into the residual stream with a recorded compression_factor of 6; a learned stop head self-terminates the loop at variable depth.
- Benchmarked on BBH (BIG-Bench Hard, cot_zeroshot, 27 subtasks, 50 items each, 1,350 total via lm-evaluation-harness 0.4.12): flexible-extract aggregate 0.94 ± 0.008, with tracking_shuffled_objects, boolean_expressions, formal_fallacies, and penguins_in_a_table at 1.00 but dyck_languages a 0.26 outlier.
- Strict-match scores read as near-zero not because the model fails, the author notes, but because BBH's strict regex expects the literal phrase "The answer is X" — which a latent-reasoning model never emits; per-subtask values carry about ±0.05–0.07 at 50 items.
- Stock vLLM cannot serve the model: DeepSeek-V4's hash MoE routing is keyed on input_ids, which vLLM's prompt_embeds path nulls out, so the addon overwrites embed_tokens output at target positions with the decoded latent and reserves a pad token id for accounting — all on the cudagraph fast path with no enforce_eager.
- Clients must opt in to thinking via chat_template_kwargs={"thinking": True} or the header x-ds4-thinking: 1 — without it, output is "garbage" — and must give the answer real token headroom, because each latent step bills one reserved accounting token and a tight max_tokens can return an empty answer with stop_reason=length.
Why it matters: Latent-reasoning prototypes have historically been unrunnable artifacts; bundling the NVFP4 backbone, the 35.7M-param head, and a forked vLLM serving runtime into one repo turns CoLaR-style reasoning from a demo into something a developer can actually `pip install` and `serve_ds4_reasoning.sh` — but only after learning two non-obvious footguns: the thinking flag and a shared token budget. The dyck_languages 0.26 score is the honest signal of where the head's compression still loses.
Ask SkimNews
