Research map / Vision-language and multimodal / Multimodal LLMs

Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AttWarp, a test-time method that uses an MLLM's cross-modal attention to rectilinearly warp the input image (enlarging query-relevant rows/columns) and re-runs the frozen model, with chained and distilled variants.

Problem

MLLMs miss small details and spatial relations in cluttered scenes, leading to errors in fine-grained perceptual grounding; existing test-time image manipulations (masking, cropping, markers) either discard context or add overhead.

Approach

Compute marginal attention profiles from cross-modal attention, convert them to CDFs, and invert them to resample the image axis-wise so high-attention rows/columns are expanded and low-attention ones compressed, keeping all pixels and a regular grid; optionally iterate the warp (AttWarp-Chains) or predict marginals with a distilled single-pass network.

Key results

• AttWarp-Chains on LLaVA improves TextVQA from 49.3 to 60.3 and DocVQA from 18.1 to 27.6.
• On Qwen, AttWarp-Chains reaches 85.9 TextVQA (+4.9) and 85.3 DocVQA (+8.0) over the base model.
• On visual-centric benchmarks with LLaVA, AttWarp-Chains improves MMVP by +2.7, BLINK by +2.9, RealWorldQA by +3.8, and MIA by +2.9.

Surprising findings

• Feature-space warping via hidden-state bias gave only +0.6% on TextVQA and was unstable, so pixel-space warping was preferred.
• Under ImageNet-C noise the method still gave gains, though the paper notes it inherits attention biases.

Limitations

• Performance depends on the quality of the underlying attention; noisy or misaligned attention limits gains.
• Warping can suppress peripheral context needed for global reasoning; size-related and misaligned-attention errors remain.
• Chained variant requires multiple MLLM passes, while the distilled variant trades some accuracy for speed.

Experimental setup

Datasets: TextVQA GQA DocVQA

Benchmarks: TextVQA GQA DocVQA POPE MMMU MMVP BLINK RealWorldQA MIA-Bench

Metrics: accuracy FID KID Mahalanobis distance pointing game accuracy

Baselines: FGVP SoM APIPrompting ViCrop

Builds on: ViCrop CLIP FiLM

Closest papers by meaning

• Token Warping Helps MLLMs Look from Nearby Viewpoints (CVPR 2026 · similarity 0.90)
• HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling (ICML 2026 · similarity 0.89)
• Seeing What Matters: A Training-Free Self-Guided Framework for Multimodal Detail Perception and Reasoning (CVPR 2026 · similarity 0.89)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.89)
• Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.89)
• Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception (ICML 2026 · similarity 0.88)