Kids outlearn AI—and we still don’t know why
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Children master language from roughly 10–30 million words of speech exposure — yet LLMs require orders of magnitude more data to become fluent, a divide cognitive scientists call the "data efficiency gap."
- Meta's Llama 3.1 was trained on 15 trillion tokens two years ago, and frontier models may now be pretraining on 10x more data, with the pool of easily available internet text potentially running dry by the 2030s, per Georgetown's Ethan Gotlieb Wilcox.
- Chomsky's 1950s "poverty of the stimulus" argument — that babies need innate grammatical knowledge to learn language from limited input — dominated US linguistics and shaped early symbolic AI, which ultimately failed and contributed to the 1970s AI winter.
- Neural networks built on the transformer architecture (BERT and GPT-2 in 2018–2019, then ChatGPT in 2022) revived language AI by showing that statistical pattern-learning from massive text could produce fluent language, contradicting the Chomskyan expectation that grammar couldn't be learned from data alone.
- Stanford's Michael C. Frank notes that training GPT-2 on 30 million words — roughly what a toddler hears before producing grammatically correct sentences — produces "a nonsense generator; you don't get a kid."
- Linguists and cognitive scientists including UC San Diego's Alex Warstadt are now building child-scale language models to test theories of acquisition and potentially create AI that learns far more efficiently — useful for training on video or serving minority-language communities.
Why it matters: With frontier models potentially exhausting the pool of easily accessible internet text by the 2030s, the data efficiency gap isn't just an academic puzzle — it's a hard ceiling on the current scale-first AI paradigm. Reverse-engineering how children learn from 100 million words could unlock training methods that don't depend on burning through ever-larger corpora.
Ask SkimNews
