Research map / Vision-language and multimodal / Multimodal LLMs
MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaVEn, a multi-granularity hybrid visual encoding framework that feeds SEED discrete visual tokens alongside text-guided pruned ViT continuous tokens into an LLM to improve multi-image reasoning.
Problem
MLLMs are designed for single-image input, and their visual encodings (long continuous ViT sequences or lossy Q-Former queries, or coarse discrete tokens alone) handle multi-image contexts poorly in both accuracy and context length.
Approach
Combine SEED discrete visual symbols (coarse semantics, added to a unified vocabulary) with continuous ViT patch features, and use the LLM's <EOS>-pooled discrete-token summary to score and keep only the top fraction (keeping ratio 0.25) of continuous patches via a trained patch selector supervised by Grounding SAM pseudo-labels.
Key results
- On DEMON zero-shot, MaVEn reaches 54.38 on Multi Image Reasoning and 30.24 on Visual Relation Inference, the top score on several tasks.
- On SEED-Bench, MaVEn scores 60.89 average vs 58.6 for LLaVA-1.5 and 46.4 for BLIP-2 (Avg. Video 42.11 vs 30.4 for Otter as cited in text).
- On single-image benchmarks, MaVEn reports VQAv2 79.1 vs 78.5 for LLaVA-1.5 and MME 1530.10 vs 1510.70, with the 0.25 keeping ratio cutting visual tokens from 576 to 144 (176 with 32 discrete tokens).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add discrete visual symbol tokens alongside continuous ViT tokens (hybrid encoding) (ablation) | MaVEn · SEED-Bench | Avg. All: 58.6 -> 60.89 | Helped |
| add discrete visual symbol tokens alongside continuous ViT tokens (hybrid encoding) (ablation) | MaVEn · DEMON Benchmark | score: 30.66 -> 39.51 | Helped |
| add continuous ViT tokens alongside discrete tokens (hybrid encoding) (ablation) | MaVEn · MMBench | score: 34.5 -> 65.21 | Helped |
| discrete-only visual encoding instead of continuous (ablation) | MaVEn · VQA and multi-image benchmarks | score: e.g. SEED 60.89 -> 43.2 | Hurt |
| text-semantics-guided patch reduction of continuous tokens with keeping ratio 0.25 (ablation) | MaVEn · single-image and multi-image benchmarks | token count and benchmark stability: 576 -> 144 continuous tokens | No clear effect |
| aggressive patch reduction with keeping ratio 0.1 (ablation) | MaVEn · multiple benchmarks | performance | Hurt |
| multi-granularity hybrid MaVEn vs LLaVA-1.5 on single-image benchmarks | MaVEn · MME | score: 1510.70 -> 1530.10 | Helped |
| hybrid MaVEn vs LLaVA-1.5 on multi-image benchmarks | MaVEn · DEMON Benchmark | score: 44.58 -> 54.38 | Helped |
| hybrid MaVEn vs LLaVA-1.5 on SEED-Bench video subset | MaVEn · SEED-Bench Avg. Video | score: 37.3 -> 42.11 | Helped |
| hybrid MaVEn vs VPG-C on DEMON Visual Relation Inference and Multi-Image Reasoning | MaVEn · DEMON Benchmark | score: VPG-C 37.50 vs MaVEn 34.63 on Multi Modal Dialogue | Mixed |
| hybrid MaVEn vs LLaVA-1.5 on MM-Vet | MaVEn · MM-Vet | score: 30.5 -> 30.4 | Hurt |
Surprising findings
- Discrete visual tokens alone perform poorly, and continuous-only encoding is also weak on multi-image tasks, yet the hybrid is best.
- Under the ablation table, the hybrid's DEMON score (39.51) differs from the main-table DEMON results, suggesting inconsistent reporting.
Limitations
- Discrete-only encoding is lossy and underperforms on fine-grained multi-image tasks.
- Patch selector pseudo-labels come from caption-guided Grounding SAM masks, which may be noisy.
- Several reported numbers are inconsistent across tables and text (e.g., MME baseline values, DEMON scores in ablation vs main table).
Experimental setup
Datasets: COCO Visual Genome RefCOCO MMC4-core LLaVA 558K LLaVA 665K CC/SUB
Benchmarks: DEMON Benchmark SEED-Bench VQAv2 GQA VizWizQA TextVQA ScienceQA MME MMBench MM-Vet
Metrics: accuracy score
Baselines: BLIP-2 mPLUG-Owl InstructBLIP LLaMA-Adapter-v2 LLaVA MiniGPT-4 LLaVA-1.5 Otter OpenFlamingo VPG-C Qwen-VL-Chat Shikra
Builds on: LLaVA-1.5 SEED Grounding SAM Vicuna
Closest papers by meaning
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.91)
- Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM (CVPR 2026 · similarity 0.90)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.90)
- MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.90)
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.89)
- Matryoshka Multimodal Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.