WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models

👋 Authors here — happy to answer questions about WorldCycle!

TL;DR: Video world models look great frame-by-frame, but ask one to walk forward and back and it never returns to where it started. We turn that failure into free supervision. WorldCycle uses reversible action cycles as a self-verifiable RL signal — no ground-truth trajectories needed — and cuts long-horizon state-returning drift by up to 44%, while boosting composite-action accuracy nearly 4×.

The problem nobody was measuring 🔍

Post-training for video world models mostly optimizes short-horizon visual quality or per-step action alignment. But the real bottleneck for long horizons is that there's no supervision signal: for an arbitrary action sequence, no ground-truth future state exists to measure accumulated error. So models drift, and we can't even tell.

The key insight 💡

Physics gives us one exact, annotation-free reference for free: a reversible action cycle must return to its starting state. Move the camera forward then back → you should see the exact same view. We show that strong baselines (WorldPlay, WorldCompass) fail this minimal check in two ways:

• Spatial closure failure — an inverse action sequence doesn't recover the starting view.
• Temporal consistency failure — the same action produces different displacements at different rollout depths.

Both are invisible to any short-horizon reward, and they compound with horizon length.

Method 🛠️

We turn closed action programs into dense supervision:

• Spatial closure reward — build mirrored frame pairs at every intermediate depth of a cycle and compare them, so each chunk becomes an independently verifiable closure check (no sparse endpoint-only signal).
• Temporal state-consistency reward — repeat cycles and compare co-indexed frames across them, penalizing drift of identical actions over time.

Together these force the model to learn actions as consistent state operators rather than memorized temporal patterns — and the same objective extends to composite actions (e.g. move-while-turning) with no ground-truth video.

CycleBench 📊

Since no existing benchmark diagnoses state-returning consistency, we release CycleBench: reversible, repeated, and composite cycles across short-/mid-/long-horizon settings — evaluating video world models as simulators, not just short-horizon generators.

Results

• Up to 44% reduction in state-returning drift.
• Nearly 4× improvement on composite-action accuracy over the baseline (which itself shows a 5× accuracy collapse on composite vs. simple actions).