113M Earthquake LLM Trained From Scratch on 2× A30 GPUs
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- nanoGPT-Seis trained a 113M-parameter earthquake-focused LLM from scratch on 2× NVIDIA A30 GPUs (48 GB each), completing the full pretraining lifecycle from blank folder through streaming inference in a single, reproducible pipeline.
- The corpus mixes ~24% earthquake/seismology text — ~20k full-text papers via Crossref+Unpaywall plus arXiv/EarthArXiv preprints and the "Earthquake Insights" Substack — with ~76% general text (Wikipedia + FineWeb-Edu), yielding ~823M train tokens at ~3.8 epochs.
- A controlled context-length A/B dropped perplexity ~11% when retraining at 4096 vs 1024 tokens for only ~26% more compute per step; at 4096 tokens, loss at positions 2048–4096 was 25% lower than at positions 0–64, showing the model conditions across long ranges.
- Adding general-text mix restored fluency at a measurable cost: the ~2.4:1 general:domain ratio cut bits/byte on general prose by 35% versus a paper-only base, but added 22% to domain bits/byte — the classic fluency-vs-specialization trade-off, kept alongside the domain-only checkpoint.
- The repository ships six reproducible stages (crawl → clean/dedup → 16k BPE tokenizer → GQA+RoPE decoder → 2-GPU DDP training → streaming inference) with throttled per-host PDF downloading, a low-yield journal abort gate, and resumable JSONL output, and the 113M checkpoint is hosted at jiazhe868/nanogpt_seis on the Hugging Face Hub.
Why it matters: This gives seismology researchers a fully transparent end-to-end pretraining reference on modest hardware — two A30 GPUs, completed in a day — with quantified findings they can build on: a 35% bits/byte cut on general prose from a Wikipedia+FineWeb-Edu mix, an 11% perplexity drop from quadrupling context length, and a 22% domain-sharpness cost for that fluency. For domain scientists evaluating small language models, the released v1 and v2 checkpoints offer a legible baseline rather than a frontier-scale answer.
