Research map / Vision-language and multimodal / Multimodal LLMs
ParGo: Bridging Vision-Language with Partial and Global Views
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ParGo, a vision-language projector mixing global and partial-view tokens with masked cross-attention and a cascaded partial block, plus a 1M detail-captioned pre-training set (ParGoCap-1M-PT) generated by GPT4-V and Gemini.
Problem
Attention-based vision-language projectors with a fixed number of visual tokens concentrate on prominent image regions and lose fine-grained details, hurting MLLM detail perception.
Approach
A Partial-Global projector where partial tokens each attend to a fixed subset of visual features and global tokens attend to all features, plus a cascaded partial perception block with a progressively widening masked self-attention so partial tokens interact across regions.
Key results
- ParGo (Vicuna-7B, 336 res.) reaches MMBench 73.7, MME 1913.1, SEED 67.3 and MM-Vet 33.5, beating Honeybee D-Abstractor by 2.9 MMB and 77.6 MME.
- Versus a Q-Former projector with the same 304 tokens, ParGo improves MME from 1591.74 to 1851.70 (+259.96) and MM-Vet from 28.3 to 33.1.
- Swapping Vicuna-7B for InternLM2-7B in ParGo gives the best ablation results (MME 1869.76, MM-Vet 37.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Partial-Global Perception block to Q-Former baseline (16 global + 288 partial tokens) (ablation) | Q-Former projector · MME | total score: +162.73 | Helped |
| add Cascaded Partial Perception block (ablation) | ParGo (PGP) projector · MME | total score: +92.23 | Helped |
| add Partial-Global Perception and CPP blocks to Q-Former baseline (ablation) | Q-Former projector · MM-Vet | score: +4.8 | Helped |
| replace 144 global tokens with 144 partial tokens (ablation) | global-token projector · MME | total score: +120.13 | Helped |
| add detailed-caption pre-training data (ParGoCap-1M-PT) to coarse pre-training (ablation) | ParGo pre-training · MME | total score: +116.84 | Helped |
| add detailed-caption pre-training data (ParGoCap-1M-PT) to coarse pre-training (ablation) | ParGo pre-training · MM-Vet | score: +0.9 | Helped |
| Partial-Global projector replaces Q-Former at matched 304 tokens | projector comparison · MME | total score: +259.96 | Helped |
| Partial-Global projector replaces Linear projector (576 tokens) with fewer tokens (304) | projector comparison · MME | total score: +123.87 | Helped |
| Partial-Global projector replaces Linear projector | projector comparison · MM-Vet | score: +0.3 | Helped |
| use partial tokens (ParGo) instead of Q-Former on detail-perception tasks | projector comparison · MME OCR / CNT subtasks | score: +32.5 OCR, +16.33 CNT vs Q-Former | Helped |
| switch LLM from Vicuna-7B to InternLM2-7B (ablation) | ParGo · MME | total score: +18.06 | Helped |
| switch LLM from Vicuna-7B to InternLM2-7B (ablation) | ParGo · MM-Vet | score: +3.9 | Helped |
Surprising findings
- A linear projector with 576 tokens outperforms the Q-Former projector with 304 tokens, indicating the attention-based projector loses information.
Limitations
- Ablations use a half-length training schedule (50k pre-training steps, 1 SFT epoch), and the paper reports only single runs without variance.
- Slightly underperforms Honeybee C-Abstractor on MM-Vet, which the authors attribute to full LLM fine-tuning in that work.
- Dataset quality relies on closed-source MLLM captions with only a light similarity-based filter against hallucination.
Experimental setup
Datasets: CC-3M SBU Caption-1M LAION-400M ParGoCap-1M-PT VQAv2 GQA OCRVQA VSR ScienceQA A-OKVQA RefCOCO Visual Genome
Benchmarks: MME MMBench SEED-Bench MM-Vet
Metrics: MME perception score MMBench accuracy SEED accuracy MM-Vet GPT-4 score
Baselines: LLaVA (v1) Shikra LLaVA-1.5 Honeybee (C-Abstractor) Honeybee (D-Abstractor) MiniGPT-4 mPLUG-Owl InstructBLIP Qwen-VL-Chat Q-Former Linear projector
Builds on: LLaVA-1.5 Honeybee Q-Former Perceiver Resampler LoRA
Closest papers by meaning
- DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.88)
- LRM-LLaVA: Overcoming the Modality Gap of Multilingual Large Language-Vision Model for Low-Resource Languages (AAAI 2025 · similarity 0.87)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.87)
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models (ICLR 2024 · similarity 0.87)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.87)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.