AirLLM runs 2.8T Kimi K3 on single 4GB GPU
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- AirLLM runs 70B LLMs on a single 4GB GPU without quantization, distillation, or pruning, and supports 405B Llama 3.1 on 8GB, DeepSeek-V3 (671B) on ~12GB, and Kimi K3 (2.8T) on under 4GB
- Kimi K3 (2.8T parameters) runs in 3.72GB of VRAM on a single RTX 6000 Ada through per-expert streaming, but requires flash-attn, a CUDA 12 torch build, and transformers 4.56.x
- v3.0 (June 2026) added FP8 model support, enabling DeepSeek-V3 on ~12GB and Qwen3-235B on ~3GB via a single AutoModel.from_pretrained(...) call
- The core trick keeps only one transformer layer on the GPU at a time, so VRAM requirements scale with layer size — not total model size — and optional 4-bit/8-bit block-wise quantization delivers up to 3x inference speedup with negligible accuracy loss
- AirLLM works across Llama 2/3/3.1/3.3/4, Qwen 1/2/2.5/3 (including MoE and FP8), DeepSeek V2/V3/R1, Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and Yi — plus MacOS and CPU inference paths
Why it matters: For hobbyists and small teams without multi-GPU rigs, AirLLM collapses the hardware floor for frontier-scale open models — a 2.8T-parameter MoE now boots on a consumer card. The trade-off, explicitly disclosed in the README, is that disk becomes the bottleneck: layer-sharded loading plus safetensor splitting demand significant storage and patience, not VRAM.
