Multimodal LLMs fall short in multiple domains, including spatial, embodied, physical, and temporal reasoning, gaps that are usually studied one at a time.
Is there a common primitive behind them? How can this primitive be learned, and how can it be reused across different tasks?
Formalized as visual transition reasoning: infer the unobserved part of a transition (s, a, s′) from its observed parts.
Causal dynamics
Forward dynamicsP(s′ | s, a)
Inverse dynamicsP(a | s, s′)
Counterfactual reasoning
Counterfactual removalP(s | a, s′)
Counterfactual substitutionP(s′aalt | a, s′, aalt)
Physical modeling
Outcome predictionP(s′ | s, aexo)
Cued predictionP(s′ | s, aq)
Temporal coherence
Temporal orderingP(s | s̃, a)
Temporal adjacencyP(s−, s+ | sref, s̃, a)
Data
Controlled transition rollouts
Both a training source and a benchmark.
✓Single-factor controlled taxonomy — Action, reasoning type, and scene varied one at a time
✓Counterfactual rollouts — Several outcomes from the same initial state
✓Error-typed distractors — Every wrong option names its error
✓Three test sets — In-distribution, held-out scenes, state perturbation
36,076 items20 scene types5 action types8 reasoning types
Training
Fullmix, and controlled subsets
11 subsets of about 2,000 items, each pairing one action type with one reasoning family, train Qwen2.5-VL-3B-Instruct with SFT, then GRPO.
Replicated across scales and model families
Research Questions and Experiments
Click each question to expand the evidence.
RQ1Do current MLLMs show a deficit in visual world modeling?Yes — the strongest of 38 frontier MLLMs reaches 65.8%, and the landscape and structure of failures are consistent across models.
Large deficit
The four strongest of 38 frontier MLLMs.
Pattern shared across model families
Eight frontier MLLMs by action type, reasoning type, and state perturbation.
Accuracy by action type, reasoning type, and state perturbation
Action type
Reasoning type
Perturbation
Accuracy (%)
Overall
Exo
Perc
Insp
Navi
Mani
Fwd
Inv
Rmv
Sub
Otm
Cue
Ord
Adj
Geo
App
GPT-5.4
65.8
62.4
61.2
58.3
68.8
81.2
83.8
81.7
64.5
76.0
81.9
85.8
57.1
27.0
62.7
95.5
Intern-S1-Pro
63.2
56.2
52.8
53.7
70.1
87.1
77.6
77.5
66.7
73.3
61.5
69.1
63.4
26.5
33.3
79.2
Qwen2.5-VL-72B
58.9
53.7
47.7
47.8
66.4
83.4
74.7
74.7
54.8
72.0
59.4
76.0
49.4
28.0
38.5
83.0
Gemma-4-31B
57.8
50.8
49.4
53.1
61.2
74.3
76.3
76.6
63.6
64.5
64.9
70.5
30.8
25.3
59.0
95.5
Human
92.3
92.8
92.8
92.6
90.6
92.8
92.8
92.8
91.1
92.2
93.3
91.1
92.8
91.7
97.3
98.7
Hover a column header for its full name.
RQ2Can visual world modeling be learned, and does it transfer?Yes — subsets of about 2,000 items each improve 22 of 26 external benchmarks, spanning spatial, embodied, physical, and temporal reasoning.
Spatial7 benchmarks · up to +27.3Embodied6 benchmarks · up to +12.5Physical5 benchmarks · up to +10.6Temporal4 benchmarks · up to +11.7
Each axis is one external benchmark. Gray: Qwen2.5-VL-3B-Instruct. Blue: gain after training on the best controlled subset for that benchmark (larger of SFT and GRPO), in percentage points on a square-root scale; hover a label for the accuracies.
The four benchmarks of static perception and general video QA do not improve.
RQ3Is visual world modeling a primitive, or just a collection of task-specific effects?A primitive — different sources of supervision build the same capability, and broad downstream gains result from it.
✕A collection of task-specific effects✓A shared primitive
A
Different sources improve the same capability? — Yes
A.1Each form of transition supervision generalizes broadly to other forms of transitions ✓, instead of being limited to improving its own form
A.2The model really learns the transition ✓, instead of just better question answering
Consistency (correlation) of different questions stemming from the same transition
A.3The gains come from real visual world modeling ✓, instead of a shortcut
The downstream gains result from that capability? — Yes
B.1The model learns a similar profile from different sources of supervision ✓, instead of a trivial collection of task-specific effects
55 / 55
pairs positive
01
All pairs of sources have positively correlated transfer: they improve largely the same downstream benchmarks.
B.2Downstream gains grow with the capability learned ✓, instead of bypassing it
Sources that improve WOVEN accuracy more also transfer more (r = 0.86 across the 11 sources).
B.3Transfer requires learning how the world really changes ✓, instead of the question format or any learnable rule
With the same question format and amount of training, the wrong rules of the perturbed world are learned on its own test set, yet only the correct transitions improve downstream benchmarks.
B.4The capability helps exactly the tasks that need it ✓, instead of making the model generally stronger
All 22 benchmarks with visual transitions improve; none of the 4 without them gains more than 2 points.
+
Further test of causality by intervention
Transition supervision can replace part of task-specific training data ✓
RQ4So what? How should we train visual world modeling — a training recipeSelect supervision by the reasoning operation it teaches, and prefer large state changes; All recipies prospectively validated on held-out tasks.
1How to prospectively choose the right dataSelect supervision by reasoning operationTo improve a downstream task, select supervision that teaches the reasoning operation the task requires, not supervision that matches its actions, scenes, or domains.
Correlation of the 11 subsets’ transfer profiles: after removing the shared component, blocks appear only when subsets are grouped by reasoning family (right), not by action, scene, domain, or visual similarity.
2How to get larger robustnessPrefer large state changes for robustnessFor robustness, prefer supervision with large state changes, such as object manipulation over camera motion.
3What are the internal boundariesSupervise temporal and agent-driven tasks directlyTransfer does not cross the boundary between the temporal family and the other three, nor the boundary between passive physical events and agent-driven actions.
4Where should we stopTransition supervision helps transition tasksFor downstream tasks that involve no transition, such as static perception, transition supervision does not help.
Prospective validation. Training data selected by the recipe outperforms alternative mixtures of the same size on multiple held-out benchmarks. See our paper for details.
Citation
If you find WOVEN useful in your research, please consider citing:
@article{fan2026woven,
title={WOVEN: Weaving Visual World Modeling into Multimodal LLMs},
author={Fan, Zheyu and Zhang, Yue and Deng, Mingkai and Wang, Kangrui and Wang, Qineng and Chen, Canyu and Hao, Jie and Fan, Xing and Guo, Chenlei and Xing, Eric P. and Bansal, Mohit and Li, Manling},
journal={arXiv preprint arXiv:2610.12417},
year={2026}
}
