Research map / Vision-language and multimodal / Multimodal LLMs
Auto-Controlled Image Perception in MLLMs via Visual Perception Tokens
Runpeng Yu, Xinyin Ma, Xinchao Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Visual Perception Tokens (DINO) | Qwen2-VL-2B · Average (max res 512) | GPT-4o judged score: 0.572 -> 0.738 | Improved |
| add Visual Perception Tokens (DINO) | Qwen2-VL-2B · Average (max res 224) | GPT-4o judged score: 0.448 -> 0.576 | Improved |
| add Visual Perception Tokens (CLIP) to 7B model | Qwen2-VL-7B · Average (max res 512) | GPT-4o judged score: 0.624 -> 0.773 | Improved |
| allow model to freely choose perception token (Free Choice) | Qwen2-VL-2B-VPT (DINO) · Average (max res 512) | GPT-4o judged score: 0.738 -> 0.749 | Improved |
| use SAM instead of DINO as re-encoder | Qwen2-VL-2B-VPT · Average (max res 512) | GPT-4o judged score: 0.738 -> 0.735 | No clear effect |
| add Visual Perception Tokens (DINO) | Qwen2-VL-2B · OpenImage (max res 512) | GPT-4o judged score: 0.418 -> 0.872 | Improved |
| add Visual Perception Tokens (DINO) | Qwen2-VL-2B · LLaVA Instruction Tuning (max res 512) | GPT-4o judged score: 0.728 -> 0.797 | Mixed |
| use region selection tokens with k=8 instead of raw bounding boxes (ablation) | Region Selection Token · DocVQA (region token ablation) | GPT-4o judged score: 0.219 -> 0.369 | Improved |
| increase region grid from k=8 to k=16 (ablation) | Region Selection Token · DocVQA (region token ablation) | GPT-4o judged score: 0.369 -> 0.307 | Hurt |
| increase region grid from k=8 to k=32 (ablation) | Region Selection Token · TextVQA | GPT-4o judged score: 0.686 -> 0.634 | Hurt |
| decrease region grid from k=8 to k=4 (ablation) | Region Selection Token · DocVQA (region token ablation) | GPT-4o judged score: 0.369 -> 0.251 | Hurt |
| add control hidden state of Re-Encoding Control token to projector (ablation) | Vision Re-Encoding Token · CUB Bird (control info ablation) | GPT-4o judged score: 0.7459 -> 0.8943 | Improved |
Closest papers by meaning
- Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.90)
- Visual Perception by Large Language Model’s Weights (NeurIPS 2024 · similarity 0.90)
- Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models (ICLR 2026 · similarity 0.90)
- Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation (NeurIPS 2025 · similarity 0.89)
- ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs (NeurIPS 2025 · similarity 0.89)
- Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
