ChatGPT-4o Plays 8-Bit Game Using 'Smart Senses'
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Russell Harper revived his 1990 8-bit game PvP-AI and ported it to the Commander X16 retro-computer emulator, where it runs at nearly 8.6 frames/s with more detail and improved AI; on actual hardware, a VERA module line-drawing bug forces a slower fallback, dropping performance to 4 fps.
- Harper connected ChatGPT-4o to the game using 'smart senses' — structured, text-based representations of the game world that abstract away heavy perception tasks, letting the LLM reason about state and plan actions instead of deciphering raw visual or audio output.
- The integration chain routes ChatGPT API calls through a PHP interface layer and a new VIA2-socket feature (currently a pull request under review) added to the x16-emulator, completing two-way communication between the cloud LLM and the retro game.
- The LLM progressed from experimentation to a winning strategy across three sequentially recorded 'ChatGPT vs PvP-AI' games, with Harper carrying persistent notes from game to game to document the arc.
- Harper made cost-driven accommodations — on-demand screen captures, removed sound and other non-essentials, and API calls every other frame instead of every frame — to keep the platform financially viable for research.
- Harper plans future research into more advanced 'smart senses' including vision, hearing, and balance capabilities for LLM-game interaction.
Why it matters: Harper's 'smart senses' approach — feeding ChatGPT-4o structured text state instead of raw pixel or audio data — offers a cheaper, simpler path to LLM game-playing than multimodal setups. The three-game arc, ending in an LLM win, suggests clean state abstractions let language models develop winning strategies in constrained game environments without expensive perception pipelines.
