Research map / Large language models / Reasoning and chain of thought

Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Masking-KD, a think-answer VLM distillation method that masks salient reasoning prefixes (token-wise, with a self-paced budget) in the student's attention so it relies more on visual evidence, outperforming VLM distillation baselines and OPSD on Qwen3-VL-Thinking models.

Problem

Distilling long think-answer traces of VLMs into compact students suffers from visual forgetting, as accumulated textual reasoning prefixes let the student imitate the teacher without using visual evidence.

Approach

Mask high-attention reasoning prefixes per next-token prediction via a salient reasoning-prefix attention mask, with a per-token masking budget scheduled by teacher-student reverse KL, so the student must draw on visual evidence during distillation.

Key results

• Masking-KD 2B student reaches 47.34 average vs 38.26 for naive response distillation and 24.12 for undistilled Qwen3-VL-2B-Thinking (8B teacher).
• Masking-KD 4B student reaches 56.66 average, surpassing the undistilled 8B teacher (53.65).
• Self-distilled Masking-KD-8B reaches 58.06 average vs 53.65 base and 55.13 for OPSD.

Surprising findings

• Masking visual or question tokens instead of response prefixes performs worse than naive distillation (37.39 and 31.31 vs 38.26).
• Masking the immediate previous token degrades average accuracy from 47.34 to 40.91.

Limitations

• Higher per-step time and memory overhead than naive distillation (157.9 s vs 75.8 s per step) due to auxiliary student forward and attention maps.
• Gains are smaller when combined with naive response distillation, attributed to conflicts with methods that explicitly transfer visual knowledge.

Experimental setup

Datasets: ViRK39K

Benchmarks: Geometry-3K MathVista We-Math MMK12 MathVerse LogicVista MMMU-Pro

Metrics: pass@1 average accuracy

Baselines: Qwen3-VL-8B-Thinking Ovis2 InternVL3.5 InternVL3 MiMo-VL-8B Qwen2.5-VL Naive response distillation LLaVA-KD CompoDistill Align-TI OPSD

Builds on: reverse KL distillation nucleus sampling (top-p) OPSD

Closest papers by meaning

• See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflection (CVPR 2026 · similarity 0.88)
• Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs (ICLR 2026 · similarity 0.88)
• Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs (AAAI 2026 · similarity 0.88)
• SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs (ICML 2026 · similarity 0.88)
• VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.88)
• SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.88)