ntransformer Runs Llama 70B on RTX 3090 via NVMe — SkimNews
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- ntransformer implements a 3‑tier adaptive caching system (VRAM, pinned RAM, NVMe/mmap) that automatically sizes layers to hardware resources.
- Llama 70B can be streamed on a single RTX 3090 (24 GB VRAM) with optional NVMe direct I/O, bypassing the CPU entirely.
- NVMe‑direct I/O achieves an 83× speedup over the mmap baseline for 70 B on consumer hardware (RTX 3090 + 48 GB RAM).
- Layer skip using cosine‑similarity calibration removes about 20 of 80 layers per token with minimal quality loss.
- GPU‑nvme‑direct backend reads model weights directly into pinned GPU‑accessible memory, overlapping NVMe reads, PCIe DMA, and compute via double‑buffered pipelines.
- Self‑speculative decoding uses VRAM‑resident layers as a draft model without requiring an extra model.
- Setup scripts modify system settings (GRUB, IOMMU, Secure Boot) and bind NVMe devices to VFIO, but warn of potential data loss and system instability if misconfigured.
Why it matters: The 83× speedup and CPU‑bypass let AI developers and research labs run a 70 B LLM on a single RTX 3090, dramatically lowering hardware expense and latency compared to multi‑GPU or CPU‑heavy setups. At the same time, the low‑level NVMe handling introduces risk of data loss if misconfigured.
Ask SkimNews
