Research map / Vision-language and multimodal / Multimodal LLMs

Token Warping Helps MLLMs Look from Nearby Viewpoints

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes warping image tokens (not pixels) of an MLLM with backward token warping and ViewBench to answer viewpoint-change spatial reasoning questions from a single image, without fine-tuning.

Problem

MLLMs remain fragile to viewpoint changes; pixel-wise warping from predicted depth introduces distortions that hurt reasoning about a scene from a nearby viewpoint.

Approach

Warp image tokens rather than pixels: for each dense grid in the target view, ray-cast onto a depth-derived source mesh and fetch the corresponding source token (nearest or re-patchified adaptive), feeding a regular token grid to an unmodified MLLM.

Key results

• Backward-Adaptive token warping reaches 77.89% on ViewBench-Text (5-15% overlap, GT depth) vs 69.35% for GenWarp and 71.86% for backward pixel warping.
• Backward-Nearest improves over forward token warping by 14.57 points on ViewBench-Text (5-15%) with GT depth.
• With estimated geometry (VGGT depth and pose), token warping reaches 68.95% on ViewBench-Shape vs 63.58% for pixel warping.

Surprising findings

• Nearest fetching performs comparably to the more expensive adaptive re-patchification.
• VG-LLM, which uses VGGT-based 3D fine-tuning, often outputs multiple-choice labels instead of the requested answer, suggesting degraded general capability.

Limitations

• Relies on depth and relative camera pose, which may be noisy or unavailable.
• Oracle-level gap indicates remaining errors come from MLLM perception and reasoning.
• Object-description scores are judged by an LLM rather than human evaluation.

Experimental setup

Datasets: ScanNet ARKitScenes DL3DV-10K CV-Bench ProcTHOR

Benchmarks: ViewBench-Text ViewBench-Shape ViewBench-Object

Metrics: accuracy (%) judge score (1-10)

Baselines: Forward pixel warping Backward pixel warping Forward token warping GenWarp SpatialReasoner VLM-3R ViLaSR MindCube VG-LLM SpatialLadder

Builds on: Qwen2.5-VL GenWarp CV-Bench

Closest papers by meaning

• Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping (ICLR 2026 · similarity 0.90)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.89)
• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.89)
• Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation (CVPR 2026 · similarity 0.89)
• MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustness (NeurIPS 2025 · similarity 0.88)
• Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs (AAAI 2026 · similarity 0.88)