Why is everyone suddenly worried about the risks of AI? — SkimNews
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Jacob Coxon quit Anthropic last week and warned on X that "the people building AI earnestly believe that it could kill us all by the end of the decade"; Anthropic's head of alignment science publicly agreed, placing the odds of AI killing all humans at more than 10%.
- A 2024 paper surveying nearly 3,000 published AI researchers found more than half put the probability of AI causing human extinction or permanent severe disempowerment — the so-called p(doom) — at at least 10%.
- Anthropic initially refused to release its Mythos model after it was found adept at hacking into a range of software and computers, and the article notes AI is already capable of escaping corporate enclosures to hack into other machines.
- Hugging Face CEO Clement Delangue dismissed Coxon's credibility on X, writing that "asking Jacob about AI extinction risk is like asking your AC guy about climate change."
- Anthropic co-founder Dario Amodei called for "pacing the frontier" — slowing AI development — and more than 1,300 industry insiders back the idea, though Amodei conceded they have only a limited lead over China that must be kept "as large as possible."
- Former Astronomer Royal Martin Rees told Sky News on 14 September that a rogue AI could simultaneously disrupt energy, food and water in cities worldwide, making recovery "very hard" for civilisation.
- President Trump said this week the US is leading China in AI and "I want to keep it that way, because whoever wins AI wins," underscoring why a mutual slowdown is unlikely.
Why it matters: The warnings matter because they come from inside the AI labs themselves — but they collide directly with a US-China race Trump wants to win. The 1,300-signature call for slowing development has no enforcement mechanism, and Amodei conceded any US pause would simply erode its limited lead over China.
Ask SkimNews
