Research map / Vision-language and multimodal / Multimodal LLMs

Detached Skip-Links and $R$-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Detached Skip-Links (stop-gradient on shallow ViT features fused into an MLLM) and R-Probe, a reconstruction-based diagnostic using an LLM-initialized shallow decoder, to improve OCR in MLLMs.

Problem

MLLMs underperform on OCR and fine-grained perception; naive multi-layer skip fusion sends high-level LLM gradients into shallow ViT layers, causing gradient interference and unstable training. Existing probes (linear classifiers) are poor at measuring fine-grained visual information transmitted to

Approach

Apply a stop-gradient to shallow skip features before concatenating them into the fusion adapter, so shallow layers receive no semantic gradients while still supplying low-level detail; and measure visual token fidelity by the pixel-reconstruction loss of a probe whose decoder is initialized from the first quarter of LLM layers.

Key results

• Detached multi-layer fusion (PE-ours) improves overall average from 61.1 to 63.0 and OCR category from 65.2 to 68.3 over the PE baseline.
• Across four ViT backbones, the method improves overall category average by +1.8 to +2.5 points (e.g., InternViT 56.2 to 58.7).
• R-Probe reconstruction steps to MSE<0.75 drop from 2158 to 1689 for the PE baseline with detached stride-6 fusion.

Surprising findings

• Adding R-Probe as an auxiliary loss slightly reduces CharXiv reasoning (43.9 to 42.8) and AI2D (72.2 to 69.2) while improving OCR benchmarks.
• R-Probe reconstruction loss with masked image regions is much higher without language tokens (1.980) than with them (1.103), indicating language-side info aids reconstruction.

Limitations

• Internal portion of training data cannot be released, limiting reproducibility.
• Auxiliary R-Probe loss trades off slightly on abstract reasoning benchmarks.
• Theoretical analysis relies on simplifying assumptions (early-phase noise dominance) that the authors note may not hold at large learning rates.

Experimental setup

Datasets: InternVL-Chat-V1-2-SFT-Data GRIT LLaVAR A-OKVQA geo170K LNQA MAVIS Screen2Words MMDU

Benchmarks: OCRBench DocVQA TextVQA ChartQA OCRVQA AI2D CharXiv

Metrics: accuracy average score per category reconstruction MSE steps to MSE<0.75

Baselines: Dense Connector (DC) DeepStack Multi-Layer Visual Feature Fusion (ML) PE-baseline

Builds on: Dense Connector DeepStack Multi-Layer Visual Feature Fusion VLMEvalKit

Closest papers by meaning

• Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.87)
• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.87)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.87)
• ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.86)
• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.86)
• Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.86)