Research map / Vision-language and multimodal / Robot action and navigation

Sparse Planning in Visual World Models via Cost Gradients

NeurIPS 2026 · Vision-language and multimodal › Robot action and navigation · approach: Embodied multimodal LLM agents

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CostGrad, a training-free goal-conditioned token selector that ranks spatial tokens by planning-cost gradient norm to sparsify CEM planning in token-based visual world models, and shows its benefit depends on the action-conditioning architecture (AdaLN vs concat).

Problem

Planning with token-based visual world models via CEM requires repeated rollouts over large spatial token grids, which is expensive; prediction-derived importance signals may not reflect control relevance.

Approach

Score each input token by the gradient norm of a one-step planning cost (distance of predicted next-state to goal) with respect to that token, keep the top-K tokens, and run CEM only on the retained subset; the selection is goal-conditioned and needs no retraining.

Key results

• At 50% tokens on AdaLN-Zero, CostGrad averages 77.3% success vs 73.5% for Full, exceeding Full on Wall (92.0 vs 84.7), PushT, and MetaWorld.
• CostGrad gives a measured 2.6x wall-clock speedup per planning step; combined with halving CEM iterations, about 5.2x total speedup at 92.0% success on Wall.
• On matched concat predictors, CostGrad has no consistent advantage over Random (Wall -0.1±4.9 pp; PointMaze +2.0±4.5 pp); action-pathway drift ratio vs Random is 0.29 on AdaLN but 1.33 on concat.

Surprising findings

• Selecting the bottom-gradient tokens collapses planning (0/60 on AdaLN, 1/96 on concat), while the top-gradient selection is strongly beneficial on AdaLN.
• Sparse planning at 50% tokens exceeds full-token planning on three of four environments, which the authors attribute to removing noisy background tokens from the cost.

Limitations

• Evaluated only on four simulated environments, one encoder (DINOv2 ViT-S/14) and one planner (CEM).
• Selector-architecture finding tested only on AdaLN vs channel concat predictors.
• Selection uses a one-step probe while CEM evaluates longer horizons.

Experimental setup

Datasets: PointMaze Wall PushT MetaWorld

Benchmarks: PointMaze Wall PushT MetaWorld

Metrics: planning success rate wall-clock time per planning step action-pathway KL drift

Baselines: Full PredAttn PredGrad Random ToMe Sparse Imagination

Builds on: DINO-WM JEPA-WMs AdaLN-Zero Sparse Imagination cross-entropy method

Closest papers by meaning

• Predictive but Not Plannable: RC-aux for Latent World Models (NeurIPS 2026 · similarity 0.89)
• World-aware Planning Narratives Enhance Large Vision-Language Model Planner (NeurIPS 2025 · similarity 0.87)
• Coarse-to-Fine Compositional Diffusion for Long-Horizon Planning (NeurIPS 2026 · similarity 0.87)
• HaM-World: Soft-Hamiltonian World Models with Selective Memory for Planning (NeurIPS 2026 · similarity 0.87)
• DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning (ICML 2025 · similarity 0.87)
• Learning Cognitive Maps from Transformer Representations for Efficient Planning in Partially Observed Environments (ICML 2024 · similarity 0.86)