Hybrid Centaur Optimizer Beats Pure LLM and Classical
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Centaur, a hybrid optimizer that shares CMA-ES's mean vector, step-size, and covariance matrix with an LLM, achieved the best results in the authors' hyperparameter tuning experiments.
- Classical methods (CMA-ES, TPE) consistently outperformed LLM-based agents within a fixed search space, where avoiding out-of-memory failures mattered more than search diversity.
- LLM agents narrowed but did not close the gap when allowed to directly edit source code, even with frontier models including Claude Opus 4.6 and Gemini 3.1 Pro Preview.
- The autoresearch repository served as the testbed, letting an LLM agent optimize hyperparameters by editing training code for a small language model under a fixed compute budget.
- A 0.8B-parameter LLM was sufficient for Centaur to outperform all classical and pure LLM methods, while unconstrained code editing required larger models to stay competitive.
- The authors' conclusion: LLMs work best as a complement to classical optimizers rather than a replacement, since LLMs struggle to track optimization state across trials while classical methods lack domain knowledge.
Why it matters: For ML teams weighing LLM agents against battle-tested optimizers, the finding that a 0.8B-parameter model inside Centaur beat both pure-LLM and classical baselines suggests hybrid pipelines can capture domain knowledge without paying for frontier-model inference on every trial.
Ask SkimNews
