An Anthropic researcher just gave us a peek at self-improving AI
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Anthropic published "Automated Researchers Can Reliably Mitigate Alignment Failures," led by fellow Chen Yueh-Han, reporting that its Automated Alignment Researcher (AAR) improved performance on all 10 targeted alignment benchmarks without degrading overall performance.
- Each AAR iteration searches the literature, proposes a method, and trains the model for 30 minutes, with effective methods preserved and ineffective ones discarded across successive iterations.
- The paper states the best AAR method beats what experienced humans propose on average within six hours, and costs roughly $4 per hour in API inference versus the $150 per hour Anthropic pays human researchers.
- Anthropic frames the result as early evidence that "automated alignment post-training could become practical in the near term," and positions the work as a concrete step toward recursive self-improvement in AI.
- The authors flag clear limitations: the automated system only works insofar as the benchmarks reflect actual alignment goals, and establishing, maintaining, and expanding both those benchmarks and the underlying literature remains significant open work.
Why it matters: The 37x cost gap — $4/hour versus $150/hour — means alignment research could scale far faster than human-led efforts, but the paper's own caveat that the system is only as good as its benchmarks means humans still define what "aligned" actually means, at least for now.
Ask SkimNews
