Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- NanoEuler is a ~116M-parameter GPT-2-class decoder-only transformer built entirely from scratch in C/CUDA, with no PyTorch, no autograd, and no ML libraries of any kind.
- Every analytic gradient is hand-derived and verified against a central finite-difference check running in double precision, with the full model passing at a max relative error of 1.02e-04.
- The CUDA engine pairs cuBLAS matmuls (TF32 tensor cores) with a hand-written tiled FlashAttention kernel using online softmax and no T×T matrix in memory, with GPU vs CPU gradients validated to ~1e-6.
- The full pipeline ships in one repo: a hand-written byte-level BPE tokenizer (4096 vocab, ~3.4 bytes/token), pretraining on Project Gutenberg + a FineWeb-Edu slice, and supervised fine-tuning on Alpaca with response-masked cross-entropy.
- FlashAttention made the GPU training step about 3× faster; a CPU showcase model (~0.76M params) trains in a few hours on 12 cores using -O3, OpenMP, and libm.
- The project name comes from the observation that the residual update x = x + f(x) is one step of the forward-Euler ODE method — a nod to Leonhard Euler.
- The author is explicit this is an educational artifact, not a product: a model of this size trained on one GPU produces 'fluent shape, shallow substance,' and DPO plus scaling toward ~270M are next on the roadmap.
Why it matters: For ML engineers and students, the value here is not a new model but a fully auditable, single-file reference implementation: every kernel, gradient, and tokenizer pass is hand-written and gradient-checked, proving that a modern transformer stack can be built and understood end-to-end without any framework in the way.
Ask SkimNews