LLM Agents Play Pokemon Showdown — SkimNews
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Oliver Grabner released a GitHub project, multi-agent-poke-showdown, that wraps the official Pokémon Showdown simulator as a Gym-style environment where four LLM-controlled Pokémon battle in 2v2 format, aiming to measure coordination rather than the single-agent skill tested by the prior PokéAgent Challenge benchmark.
- Free-talking teammates won 64% of their battles against a scripted max-damage bot, compared with 42% for teammates who could not talk, and even a single one-message exchange (52%) beat silence, according to seeded evaluations using Qwen3.8-27B on both sides.
- One model controlling both Pokémon won 54% of battles, better than no talk but worse than two agents talking freely, leading the author to attribute the gain to the conversation itself rather than a single mind planning both moves.
- Free talk consumed 978k prompt tokens per battle, roughly 3.8x the 259k used in no-talk mode, because every message triggers a fresh model call that re-reads the battle state and the prior chat.
- The environment discriminates between model strengths: a Qwen3.8-27B team beat a 2B-model team 96% of the time, a 9B team 74% of the time, and the max-damage bot 64% of the time, with closer games against stronger opponents.
- Grabner plans to isolate the effect of talking from the effect of seeing a partner's chosen move, convert the environment into a cross-model leaderboard benchmark, and use it for reinforcement-learning training aimed at collaborative behaviors.
- The experiments ran on Texas A&M's Grace cluster, with Grabner thanking advisor Prof. James Caverlee and Maria Teleki for guidance and feedback.
Why it matters: The project gives multi-agent LLM researchers a reproducible benchmark where coordination, not raw battling skill, is the variable being measured, and it comes with a concrete efficiency bill: a 22-point win-rate lift costs nearly 4x the tokens, a trade-off any team shipping multi-agent systems will have to price in.
Ask SkimNews
