AMD Runs 1‑Trillion‑Parameter LLM on Ryzen AI Cluster
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- AMD demonstrates a four‑node cluster of Framework Desktop Ryzen AI Max+ 395 systems (each with 128 GB RAM) can run the 1‑trillion‑parameter Kimi K2.5 model locally using llama.cpp RPC.
- Kimi K2.5 is a 375 GB open‑source model from Moonshot AI, tuned for coding, long‑horizon reasoning, and multimodal inputs.
- Framework Desktop nodes extend per‑node VRAM from the BIOS‑limited 96 GB to 120 GB via TTM kernel parameters, achieving a total of 480 GB GPU‑addressable memory across the cluster.
- Lemonade SDK provides pre‑built ROCm‑enabled llama.cpp binaries for the gfx1151 GPU, simplifying setup; alternatively, llama.cpp can be built manually with ROCm 7.0.2 and RPC support.
- ROCm 7.0.2 drivers and the llama.cpp RPC engine enable the four machines to act as a single logical accelerator over a 5 Gbps Ethernet interconnect.
Why it matters: Developers and researchers can now run a state‑of‑the‑art 1‑trillion‑parameter model on a modest on‑premise AMD cluster, cutting reliance on expensive cloud GPUs and lowering inference costs while retaining multimodal capabilities. The guide shows how to boost per‑node GPU memory to 120 GB, delivering 480 GB across four nodes, making local inference feasible for enterprises and hobbyists.
