Resolving State-Representation Mismatch: State-Space Visual Reasoning for Open-Loop VLA Planning

Anonymous authors

SSVR decouples static visual context from an action-conditioned recurrent latent state for open-loop planning.

MazeFlip: Dynamic Observation Frames New experiment

To further investigate whether SSVR can be applied to dynamic‑map scenarios, we construct MazeFlip based on the Maze dataset. Specifically, we introduce dynamic properties to original Maze maps: LEFT/RIGHT actions trigger vertical flipping of the subsequent observation view, while UP/DOWN actions trigger horizontal flipping. The model only takes the initial map as input and is required to infer, via internal latent evolution (“mental simulation”), the correct optimal action sequence for reaching the destination under these dynamic rules.

Quantitative results

After five epochs, SSVR reaches 86.20% EM and 94.82% PR, with a 99.90% legal-action rate.

Exact match (EM)Progress rate (PR)

MazeFlip evaluation across five training epochs. Scores are percentages.

MazeFlip training evaluation. Higher is better for Legal, Closer, EM, and PR.
Epoch	Step	Legal ↑	Closer ↑	EM ↑	PR ↑	Entropy
1	1,092	97.30%	83.20%	33.70%	54.55%	0.5601
2	2,184	99.60%	98.10%	60.50%	80.02%	0.4556
3	3,276	100.00%	98.80%	72.00%	88.58%	0.4712
4	4,368	100.00%	99.80%	80.40%	92.25%	0.4940
5	5,460	99.90%	99.40%	86.20%	94.82%	0.5002

Legal: valid-action rate. Closer: distance-reducing action rate. EM: exact match. PR: progress rate. Entropy is unitless. Download CSV.

Qualitative results

MazeFlip example 1

MazeFlip level 4: a five-action rollout with animated vertical and horizontal flips.

Trajectory

MazeFlip example 2

MazeFlip level 6: a seven-action rollout that tracks orientation across successive moves.

Trajectory

Yellow outlines mark the agent position before each action. The map and attention overlay are reflected together for visualization; flip animations do not represent additional observations supplied to the model. Use the selectors to view all eight MazeFlip cases.

Abstract

Despite rapid progress in vision-language-action (VLA) models, existing reasoning paradigms still face a fundamental state-representation mismatch in open-loop planning. Given only an initial observation, models must internally simulate action-conditioned state transitions, whereas text-, pixel-, and latent-space reasoning can suffer from lossy spatial compression, error-accumulating visual generation, and bypass of intermediate latent tokens, respectively, undermining reliable long-horizon planning.

We propose State-Space Visual Reasoning (SSVR), which decouples static visual context, language constraints, and a recurrent latent state. SSVR encodes the initial image and instruction once, then conditions each action prediction on the latent state and updates it with an action-conditioned GRU. Using Qwen2.5-VL as the backbone, SSVR achieves 99.5/99.6, 96.3/98.0, and 83.9/90.6 EM/PR on FrozenLake, Maze, and MiniBehavior, substantially outperforming prior methods. Extensive experiments support the effectiveness of recurrent state modeling for VLA open-loop planning across input transformations and transfer settings.

By reusing static visual-textual context and updating a compact recurrent state, SSVR supports efficient multi-step inference, achieving up to 98.58× faster Maze decoding rollouts than the evaluated baselines with the prefix cache prebuilt.

Method

SSVR encodes the initial image and instruction into reusable visual-textual context. At each step, a latent state token conditions action prediction through the pretrained language-model head. An action-conditioned GRU then updates this state for the next decision.

SSVR architecture: cached text and image embeddings are reused at each step; a state token conditions the MLLM action prediction and is updated recurrently. — **Figure 1.** Initial visual and language context is cached. Each latent state conditions the next action; the selected action drives the recurrent update.

z_t+1 = LN(GRU(E^a(a_t), z_t))

The backbone uses Qwen2.5-VL with LoRA adaptation. Soft-target supervision balances safe and optimal actions; the main model uses α = 0.7.

Experimental Results

Open-loop planning

SSVR achieves the highest EM and PR among the evaluated methods on all three tasks. Average EM/PR improves over VPRL by 12.6/11.2 percentage points.

Qwen SFTVPRLSSVR

FrozenLake

Risk-aware navigation

Qwen SFT68.6

VPRL91.6

SSVR99.5

Maze

Topological path finding

Qwen SFT60.9

VPRL74.5

SSVR96.3

MiniBehavior

Phased object interaction

Qwen SFT31.3

VPRL75.8

SSVR83.9

Main results: EM / PR (%). * Official releases without training on our datasets.
Method	Paradigm	FrozenLake	Maze	MiniBehavior	Average
Gemini 2.0 Flash Direct*	Text	21.2 / 47.6	8.3 / 31.4	0.7 / 29.8	10.1 / 36.3
Gemini 2.0 Flash CoT*	Text	27.6 / 52.5	6.9 / 29.8	4.0 / 31.2	12.8 / 37.8
Gemini 2.5 Pro*	Text	72.0 / 85.0	21.5 / 35.5	37.6 / 59.9	43.7 / 60.1
Qwen2.5-VL-7B Direct*	Text	1.2 / 15.0	0.6 / 14.5	0.3 / 9.8	0.7 / 13.1
Qwen2.5-VL-7B CoT*	Text	8.2 / 29.1	2.3 / 15.2	0.5 / 14.7	3.7 / 19.7
Qwen2.5-VL-7B SFT	Text	68.6 / 84.4	60.9 / 70.3	31.3 / 56.1	53.6 / 69.9
VPFT	Pixel	75.4 / 79.5	59.0 / 64.0	33.8 / 52.2	56.1 / 65.2
VPRL	Pixel	91.6 / 93.2	74.5 / 77.6	75.8 / 83.8	80.6 / 84.9
LVR*	Latent	6.8 / 16.0	5.0 / 10.2	0.0 / 8.0	3.9 / 11.3
Monet*	Latent	5.0 / 13.0	2.3 / 5.0	0.0 / 7.5	2.4 / 8.5
SSVR (ours)	Recurrent state	99.5 / 99.6	96.3 / 98.0	83.9 / 90.6	93.2 / 96.1

Inference efficiency

With a prebuilt prefix cache, SSVR decodes a Maze rollout in 182.442 ms, yielding a 98.58× speedup over VPRL under the reported timing protocol.

Maze decoding-rollout latency; 20 repetitions after warm-up.
Method	Rollout latency (ms)	Relative latency
SSVR + cache	182.442	1.00×
SSVR	356.634	1.95×
LVR	1,059.193	5.81×
Monet	6,975.866	38.24×
VPRL	17,984.442	98.58×

All methods use the same Maze case. Latency excludes preprocessing and prefix-cache construction; this is not an end-to-end timing comparison.

Generalization and VQA retention

FrozenLake → Maze: the full FrozenLake-trained model reaches 33.6% Maze EM, compared with 6.8% for VPRL.
Native spatial VQA: optimal-action hit rate increases from 23.8% to 46.7% (+22.9 pp), with recurrent state conditioning disabled.
VQAv2: SSVR retains 68.0% accuracy versus 71.9% for native Qwen (−3.9 pp).

Image-scale robustness

Performance remains close to baseline under image enlargement. Downscaling affects harder maps more strongly, consistent with sensitivity to loss of fine spatial detail.

Paper figure: Maze exact match and progress rate versus image scale from 0.7 to 1.3, overall and at levels 3 through 6. Harder levels are more sensitive to downscaling. — Original quantitative figure · image scale robustness on Maze

The scale experiments report a 96.4% Maze EM baseline; the main comparison reports 96.3%. Each result is retained under its original experimental setting.

Qualitative Results

Action-specific visual attribution evolves with the recurrent state. Select a trajectory to inspect navigation and object-interaction decisions.

FrozenLake

Avoid the holes

Trajectory

Maze

Navigate the walls

Trajectory

MiniBehavior

Navigate · pick · drop

Trajectory

Yellow outlines indicate the current position before the action; original sprites mark initial locations. All 31 successful cases contain at least three actions (including PICK/DROP); the eight MazeFlip cases appear above.