Interactive video worlds
Let World Engines Control
and Video Generators Render
1 University of British Columbia 2 Huawei Canada 3 Vector Institute for AI 4 Canada CIFAR AI Chair
You define the rules.
The engine controls what happens.
The video model brings it to life.
01 / How it works
The engine owns the rules and state.
The renderer supplies appearance, lighting, and visual detail.
A coding agent turns your interaction rules into executable logic.
if nurture_complete:
flower.status = "open"
# the engine retains the stateOnly the information needed to place, identify, and update each object.
A causal video model renders each new state in the context of its history.
Inside the renderer
Frame-wise masked cross-attention routes an object’s state to the visual regions covered by its box. Persistent attributes describe what it is; dynamic status describes what is happening now.
Read the methodLearn spatial grounding from automatically annotated open-domain videos.
Adapt to object transitions using gameplay videos and clean visual history.
Distill on self-generated histories for coherent, few-step streaming generation.
ResultsOn PairedStateBench, both branches of a counterfactual pair must show their intended outcome—and nothing happens when the condition is not met.
Paired counterfactual success
Complete-task success
Throughput · 832 × 480, 4 steps
02 / Key examples
Each pair runs the same rules on two different inputs, and the engine decides the outcome. The last example keeps the input and changes the world. Box states and held keys are shown with every video.
01 Input timing
Stay near the lotus and hold D for two seconds: it opens. Release early and the charge resets—the flower stays closed.
Engine rulenearby + hold D for 2 s → bloom
02 Conditions
With the key, pressing D at the door opens it. Without the key, the same press is rejected and the door stays locked.
Engine rulehas key + D at the door → open
03 Contact
An aligned grip attaches the cargo, so the robot can carry it and set it down. Raise the arms too high and the grip closes on air; the cargo stays on the table.
Engine rulehands on cargo + grip → attached
04 Appearance
One engine replay, rendered as a copper drone in a greenhouse, a bee spirit in a stone hall and a ruby orb among ancient columns. Only the object and scene descriptions change.
Engine rulesame box states + new descriptions → new world
03 / Gallery
35 generated videos, each driven by box states from a world engine. Click any video to enlarge it.
Explore the research
@misc{wu2026boxworldletworldengines,
title={BoxWorld: Let World Engines Control and Video Generators Render},
author={Zike Wu and Ahmed Hatem and Chintan Trivedi and Wallace Michel Pinto Lira and Xuebin Qin and Lele Wang and Renjie Liao},
year={2026},
eprint={2610.36371},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2610.36371},
}