Research map / Label-efficient and robust learning / Knowledge distillation
Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation
Jaehyun Jang, Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Mark Hasegawa-Johnson, Chang D. Yoo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| supervise response-to-vision instead of prompt-to-vision attention (ablation) | TRAG · Qwen2.5-0.5B | VQA/CR Avg: VQA 58.6 -> 60.2; CR 69.2 -> 75.2 | Improved |
| add prompt tokens to response supervision (ablation) | TRAG · Qwen2.5-0.5B | VQA Avg: 58.4 vs 58.6 (prompt only) | No clear effect |
| token-level response supervision instead of aggregated (ablation) | TRAG · Qwen2.5-0.5B | VQA/CR Avg: VQA 57.7 -> 60.2; CR 70.4 -> 75.2 | Improved |
| entropy-weighted forward/reverse KL instead of fixed measures (ablation) | TRAG · Qwen2.5-0.5B | VQA/CR Avg: VQA 58.7 -> 60.2; CR 74.4 -> 75.2 | Improved |
| use intermediate layers [0.3,0.6] instead of early layers (ablation) | TRAG · Qwen2.5-0.5B | VQA/CR Avg: VQA 58.7 -> 60.2; CR 69.2 -> 75.2 | Improved |
| shift aggregated layer range to [0.4,0.7] (ablation) | TRAG · Qwen2.5-0.5B | VQA/CR Avg: VQA 60.2 -> 60.3; CR 75.2 -> 75.0 | No clear effect |
| use larger teacher (4B instead of 1.8B) (ablation) | TRAG · Qwen1.5-0.5B student | CR Avg: 63.0 -> 70.3 | Improved |
| apply TRAG over PT-SFT on Qwen2.5-0.5B (ablation) | TRAG · VQA Avg6 | Avg6: 62.4 -> 65.3 | Improved |
| TRAG versus LLaVA-KD on Qwen2.5-0.5B | TRAG · VQA | Avg8: 58.7 -> 60.2 | Improved |
| TRAG on MobileLLaMA 1.4B student | TRAG · MobileLLaMA | VQA/CR Avg: VQA 55.2 -> 58.2; CR 50.0 -> 57.3 | Improved |
| TRAG on Qwen2.5-1.5B student on SugarCrepe | TRAG · SugarCrepe | accuracy: 87.9 -> 87.3 SugarCrepe; avg 83.8 -> 85.0 | Mixed |
Closest papers by meaning
- Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.90)
- Visual-Advantage On-Policy Distillation for Vision-Language Models (NeurIPS 2026 · similarity 0.89)
- Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.89)
- Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions (ICML 2026 · similarity 0.89)
- No Head Left Behind – Multi-Head Alignment Distillation for Transformers (AAAI 2024 · similarity 0.89)
- Towards Long-window Anchoring in Vision-Language Model Distillation (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
