Research map / Vision-language and multimodal / Multimodal LLMs

Masking Teacher and Reinforcing Student for Distilling Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Masters distills large VLM teachers into compact students by magnitude-masking teacher weights and progressively restoring them, training on multi-response data from masked teachers and the student, with offline GRPO-style RL using accuracy (LLM-as-judge) and distillation rewards.

Problem

Knowledge distillation from large VLM teachers to small student VLMs is unstable because of the large parameter/capacity gap between teacher and student, and SFT labels from huge closed-source models are too rich for small students to absorb.

Approach

Temporarily magnitude-mask the teacher's weights (per layer) and progressively restore them during distillation so the teacher's capacity matches the student's; train on pre-generated multi-response data from masked teachers and the student, and refine with offline RL combining an LLM-judged accuracy reward and a normalized distillation reward.

Key results

• Masters on InternVL3.5-8B reaches 80.0 average across 13 benchmarks vs 75.4 for the base InternVL3.5-8B and 76.8 for naive large-teacher distillation (ablation table).
• Qwen2.5-VL-7B-Masters averages 81.7 on MM-Vet and 95.6 on ChartQA, exceeding its base (71.8 MM-Vet, 87.3 ChartQA).
• Gradual teacher scaling (14B+38B+78B) gives 82.3 MathVista and 74.0 MMMU for InternVL3-8B student, beating single 38B (73.4 MathVista, 65.1 MMMU) and 78B teachers.

Surprising findings

• Masking the teacher with a max ratio of 0.2 works best; higher ratios (0.3-0.5) collapse performance for several teachers, e.g. 48.2 avg at r_max=0.5 for Qwen2.5-VL-72B.
• Using only teacher-generated or only student-generated responses is worse than a balanced 1:1 mix.

Limitations

• Offline RL on pre-generated responses limits adaptability to real-time feedback; online training is computationally infeasible (over 30 days on 256 A100s for one model per authors).
• Accuracy reward relies on an LLM judge whose quality is not separately validated.
• Reported improvements are over the teacher-based baselines the authors built; comparisons with closed-source models use reported numbers from other sources.

Experimental setup

Datasets: COCO-ReM iNaturalist2018 VQA-v2 Super-CLEVR MAVIS Geometry3K ScienceQA AI2D SA-1B LLaVAR ChartQA DocVQA

Benchmarks: AI2D ChartQA MathVista MMBench MMBench-CN MM-Vet MMMU MMMU-Pro MMStar BLINK SEED SEED2-Plus

Metrics: accuracy (%) average score across benchmarks

Baselines: DistiLLM LLaVA-KD VLsI RIL InternVL3.5-38B Qwen2.5-VL-72B GPT-4o Claude-3.7-Sonnet Gemini-2.5-Pro

Builds on: GRPO Jensen-Shannon divergence distillation magnitude-based network pruning DistiLLM LLaVA-KD VLsI RIL

Closest papers by meaning

• Visual-Advantage On-Policy Distillation for Vision-Language Models (NeurIPS 2026 · similarity 0.89)
• Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs (AAAI 2026 · similarity 0.89)
• Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• Unified Reinforcement and Imitation Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
• Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.88)
• TTRV: Test-Time Reinforcement Learning for Vision Language Models (CVPR 2026 · similarity 0.88)