Research map / Vision-language and multimodal / Multimodal LLMs

Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GAP, a granular alignment recipe for visual latent reasoning in MLLMs that maps decoder states through a PCA-aligned latent head back into the vision-embedding subspace, with context-grounded latent supervision and difficulty-aware latent assignment, built on Qwen2.5-VL-7B.

Problem

Output-as-input visual latent reasoning in pre-norm MLLMs yields unstable gains because final decoder hidden states are far from the input vision-embedding distribution (norm and subspace mismatch), and latent supervision is applied indiscriminately.

Approach

Re-inject visual latents only after a PCA-aligned latent head predicts coefficients in the principal subspace of auxiliary-image vision embeddings (applied to RMS-normalized decoder states), combined with curated context-grounded latent data and supervising latents only on queries the base model fails (8-sample accuracy = 0).

Key results

• GAP (LH+PCA+DA) reaches 73.25 on HRBench4K, 63.40 on MMStar and 71.00 on MathVista (three-seed means), versus 68.12, 60.46 and 68.50 for Qwen2.5-VL-7B.
• GAP improves Avg-P by +1.74 and Avg-R by +5.98 over Monet-7B; Monet's Avg-R drops to 47.99 from 52.62 for the base model.
• Training-free EMA norm calibration on Monet-7B improves MathVista from 61.30 to 63.30 (+2.00).

Surprising findings

• /invoke

Limitations

• Only validated on Qwen2.5-VL-7B and Monet-7B; cross-backbone generality untested.
• Monet and LVR are compared as prior systems rather than retrained on the same 49K data; difficulty threshold tau=0 is heuristic.
• Inference latency rises about 1.61x over the dense caption SFT baseline; random-basis and matched-parameter controls remain incomplete.

Experimental setup

Datasets: Monet SFT data (125K, comparison only) curated 49K latent-supervision set

Benchmarks: HRBench4K MMStar MME-RealWorld-Lite MathVista WeMath POPE

Metrics: accuracy Avg-P Avg-R Avg-3 Avg-2 RelMSE latency

Baselines: Qwen2.5-VL 7B zero-shot Dense Caption SFT Monet-7B LVR Ours-LH Ours-LH+DA

Builds on: Monet LVR COCONUT

Closest papers by meaning

• Latent Visual Reasoning (ICLR 2026 · similarity 0.93)
• SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.92)
• Monet: Reasoning in Latent Visual Space Beyond Image and Language (CVPR 2026 · similarity 0.92)
• Vision-aligned Latent Reasoning for Multi-modal Large Language Model (ICML 2026 · similarity 0.92)
• The Alignment Illusion in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.91)
• Visual Enhanced Depth Scaling for Multimodal Latent Reasoning (NeurIPS 2026 · similarity 0.91)