Microgpt 4000‑Parameter Network Visualized in Browser
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Microgpt builds its input representation by adding token embeddings (Wtok) to positional embeddings (Wpos) for each token i.
- Microgpt applies RMSnorm to the input vector before feeding it into the first transformer block.
- Microgpt uses a transformer block with two attention heads; each head computes queries (WQ), keys (WK), and values (WV) and applies softmax to obtain attention weights.
- Microgpt adds a residual connection (R1) after the attention heads and passes the result through a second RMSnorm.
- Microgpt processes the normalized vector through a feed‑forward MLP with a 64‑dimensional hidden layer (W1 → ReLU → W2) and adds another residual connection (R2).
- Microgpt generates final logits with a language‑model head (Wlm), applies softmax, and samples the next token from the resulting probability distribution.
Why it matters: The browser‑based demo lets developers and educators explore Microgpt’s full inference pipeline—including embeddings, RMSnorm, attention, and MLP layers—without installing external software, lowering the barrier to studying small language models and facilitating rapid prototyping. It also reveals the exact query, key, and value matrices for each head, giving transparent insight into attention weight computation and enabling debugging of token‑level behavior.
Ask SkimNews