arxiv:2608.08160

Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives

Published on Aug 8

· Submitted by

Authors:

Abstract

The study introduces a benchmark and formalizes narrative commitment preservation to evaluate long-horizon logical consistency in interactive storytelling with large language models.

Generated by thinkingmachines/Inkling-Small

The rapid advancement of Large Language Models (LLMs) is revolutionizing AI for Games by enabling open-ended and fluid interactive storytelling. However, existing research has largely overlooked the critical challenge of maintaining long-horizon logical consistency and narrative integrity against unconstrained user interventions. To address this, we formulate this challenge as Narrative Commitment Preservation (NCP), and take interactive narrative as our testbed. We introduce NCP-Bench, a benchmark of 100 narrative environments derived from movie synopses. Each environment includes a structured narrative specification (trajectory, commitments, and initial facts) that we can automatically check throughout the interaction between the player agent and the narrator agent. Experiments across state-of-the-art LLMs reveal a substantial long-horizon consistency gap: high linguistic quality does not guarantee commitment preservation; even strong models frequently generate logically conflicting content under adversarial interventions, with the best-performing model (GPT-5.2) achieving only 42% survival rate after 20 turns and fact conflict rates ranging from 40% to 68% across models, and only isolated runs satisfying all achievement commitments within the 100-turn limit.

View arXiv page View PDF GitHub 25 Add to collection

Community

mayingpeng

1 day ago

We release our data, code and prompt templates in https://github.com/yingpengma/NCP-Bench.

Elliott

Paper submitter about 13 hours ago

Sharing our new work at ICML 2026 on interactive narrative and long-horizon consistency ☀️ We explore the question: when players can act freely in an open-ended way, as interactions get longer and events accumulate, can the story world still hold together?

For example, the story has already established that “the only key is on the guard.” The player might choose to steal it, break down the door by force, or lie that they already have the key. The model needs to respond reasonably to the player’s actions, while ensuring that established facts do not conflict with the next turn and that later must-happen plot points do not vanish into thin air. We call this ability Narrative Commitment Preservation: preserving established facts and completing the key plot points that must be advanced later.

To this end, we built NCP-Bench from 100 classic movie synopses. Each story is designed as an interactive environment containing an initial set of facts, key events during the story, and plot goals to achieve. On NCP-Bench, Narrator Agent and Player Agent interact for up to 100 turns, and we check at each turn for contradictions, whether the model genuinely responds to the player’s actions, and whether key plot points are completed.

Our evaluation shows that long-range interaction remains difficult for current models: the best-performing model, GPT-5.2, remains conflict-free in only 42% of cases after 20 turns; across models, 40% to 68% of interactions ultimately end in factual conflicts. Under the 100-turn limit, the number of cases that make it through the entire interaction without problems while completing all key plot goals is in the single digits.