ntransformer Runs Llama 70B on RTX 3090 via NVMe
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- ntransformer implements a 3‑tier adaptive caching system (VRAM, pinned RAM, NVMe/mmap) that automatically sizes layers to hardware resources.
- Llama 70B can be streamed on a single RTX 3090 (24 GB VRAM) with optional NVMe direct I/O, bypassing the CPU entirely.
- NVMe‑direct I/O achieves an 83× speedup over the mmap baseline for 70 B on consumer hardware (RTX 3090 + 48 GB RAM).
- Layer skip using cosine‑similarity calibration removes about 20 of 80 layers per token with minimal quality loss.
- GPU‑nvme‑direct backend reads model weights directly into pinned GPU‑accessible memory, overlapping NVMe reads, PCIe DMA, and compute via double‑buffered pipelines.
- Self‑speculative decoding uses VRAM‑resident layers as a draft model without requiring an extra model.
- Setup scripts modify system settings (GRUB, IOMMU, Secure Boot) and bind NVMe devices to VFIO, but warn of potential data loss and system instability if misconfigured.
Why it matters: The 83× speedup and CPU‑bypass let AI developers and research labs run a 70 B LLM on a single RTX 3090, dramatically lowering hardware expense and latency compared to multi‑GPU or CPU‑heavy setups. At the same time, the low‑level NVMe handling introduces risk of data loss if misconfigured.
