arxiv:2610.12341

Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition

Published on Oct 8

· Submitted by

Qingle Liu on Oct 12

THU-CST-SAST

Upvote

Authors:

Qingle Liu ,

Kejin Wang ,

Yicheng Zhao ,

Jieming Li ,

Shenghan Zheng ,

Ruize Yang ,

Bojun Yang ,

Xiang Gao ,

Kaiyu Zhong ,

Chengxi Li ,

Yong Yan ,

Shengyuan Wang

Abstract

Adversarial games have driven advances from heuristic search to reinforcement learning, yet learning and adapting strategies from limited samples remain challenging. AI agents offer an alternative by turning game experience into revisions of executable policies. Building on heuristic learning (HL), we formalize Adversarial Heuristic Learning (AHL), a paradigm that uses AI agents as learning engines to refine game policies and supporting software while keeping model weights fixed. We introduce AAArena, a benchmark comprising 12 authentic adversarial games and 1,920 archived human programs, with an evaluation protocol modeled on real-world game competitions. Agents interpret rules, choose opponents, analyze replays, and revise game agents to achieve their highest ranking within fixed match and evaluation budgets. We evaluate completedmodels model and harness configurations: Opus5.5 with Claude Code earns 6 gold medals, while no evaluated configuration tops the remaining 6 human ladders. Performance is generally weaker in games with more complex rule specifications. Further experiments show that opponent selection and dense feedback support policy improvement, and that agents learn from both on-policy replays of their own matches and off-policy replays of other players' matches. These results highlight HL's potential in adversarial games and identify persistent challenges in game understanding, strategy implementation, and long-horizon policy development.

View arXiv page View PDF Project page GitHub 40 Add to collection

Community

Aoraku

Paper author Paper submitter 1 day ago

In Learning Beyond Gradients, Jiayi Weng (@trinkle23897) discusses Heuristic Learning (HL): an agent acts as the learner, using experience and feedback to directly improve executable structures such as programs.
Looking back, isn't every contestant's development process a form of heuristic learning? Understand the rules, write a strategy, test it, and revisit losses. Find a weakness and turn an insight into code. The contestant is the learning engine, converting experience into a stronger program.

Following this idea, we replace the human learning engine with an AI agent and formalize HL in adversarial tasks as Adversarial Heuristic Learning (AHL). The foundation model's weights stay fixed. The agent understands the rules, selects opponents, analyzes feedback, and generates and maintains the game-playing agent that enters each match.

Agents could test more persistently, organize replays systematically, and draw inspiration from different players. But whether these advantages lead to victories requires actual competition.

To investigate, we built AAArena, drawing on the contest's archives to collect 12 historical competition games and 1,920 human-written program submissions. Agents compete for leaderboard positions through the same basic process as human players: read the rules, play matches, and improve the code.

To give different agents a common evaluation budget, each game provides 128 units for small matches and 16 evaluations against the full opponent pool. Small matches let the agent select opponents, compete, and inspect replays. Full evaluations return the current strategy's Elo rating and rank. Agents decide when to learn and when to submit; comparisons use their best official result achieved within the budget.

Across seven AI models and their runtime configurations, Opus5.5 with Claude Code earned six gold medals. No tested configuration topped the other six leaderboards. Agents can already match strong human competitors in some games; the remaining games reveal where their abilities fall short.