Research map / Label-efficient and robust learning / Knowledge distillation

DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Few-shot and metric distillation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that up to 69% of chart/document reasoning prompts are zero-delta for teacher-student distillation, and synthesizes 200k high-divergence prompts (DeltaPrompts) via seed-guided and skill-guided teacher generation with answer-divergence rejection, yielding gains for Qwen3-VL-8B-Thinking and transfer to GLM-4.1V.

Problem

Distillation of compact VLMs relies on prompts chosen heuristically or from off-the-shelf datasets, but many prompts give the teacher and student identical answer distributions, so they provide little learning signal and performance saturates with data scale.

Approach

Defines answer divergence (reverse KL over LLM-judge-grouped final-answer distributions from K=16 rollouts) as a per-prompt measure of teacher-student capability gap, and uses the teacher as a data generator with seed-guided and skill-guided (student-failure-mode) synthesis followed by rejection of zero-delta prompts.

Key results

• On-policy distillation with DeltaPrompts yields Qwen-8B-DeltaThinker average 72.89 on chart/document benchmarks vs 67.92 for Qwen3-VL-8B-Thinking, and 54.12 vs 46.92 on real-world perception benchmarks.
• Up to 69% of prompts in existing chart/document datasets are zero-delta; delta-filtered subsets reduce E-Vendi diversity by up to 29.4%.
• Transfer to GLM-4.1V-9B-Thinking with GLM-4.5V teacher, reusing 154k retained prompts, gives GLM-9B-DeltaThinker average 73.06 on chart/document benchmarks.

Surprising findings

• Up to 69% of prompts in standard chart/document datasets are zero-delta, meaning teacher and student produce identical answer distributions.
• Filtering existing datasets for non-zero divergence reduces diversity, so delta prompts show lower E-Vendi than random or zero-delta subsets.

Limitations

• Divergence is measured once against the initial teacher-student pair and the dataset is fixed, so divergence may drift as the student evolves.
• Images are sourced from existing datasets rather than synthesized.
• Evaluation focuses on visual chart, document and perception domains; generality to long-horizon multimodal tasks is untested.

Experimental setup

Datasets: ChartQA InfoVQA arXivQA SciVQA DeepEyes47k DriveAction VisualProbe train PDF-VQA Div2k Flickr2k sama-drives-california

Benchmarks: CharXiv InfoVQA ChartQAPro SEEDBench2-Plus RealWorldQA HRBench 4K V* VisualProbe MathVista LogicVista MMMU-Pro

Metrics: accuracy average accuracy standard error over 5 runs E-Vendi diversity answer divergence

Baselines: Qwen3-VL-8B-Thinking GLM-4.1V-9B-Thinking Bee-8B-RL REVisual-R1 Vision-R1-7B ARES-RL-7B SFT on existing mixture

Builds on: on-policy distillation verl vllm E-Vendi Jung et al. balanced sampling

Closest papers by meaning

• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.88)
• Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models (CVPR 2026 · similarity 0.88)
• Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale (ICML 2026 · similarity 0.88)
• Visual-Advantage On-Policy Distillation for Vision-Language Models (NeurIPS 2026 · similarity 0.88)
• Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View (AAAI 2026 · similarity 0.87)
• VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation (CVPR 2026 · similarity 0.87)