Research map / Vision-language and multimodal / Multimodal LLMs

CORE: Compact Object-centric REpresentations as a New Paradigm for Token Merging in LVLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CORE, which uses a shared ConvNeXt-L encoder with a Mask2Former head to merge visual tokens into one token per object, sorts them by mask centroid, and feeds them to a LLaVA-NeXT-style InternLM2-7B LVLM for efficient fixed- and adaptive-rate compression.

Problem

LVLMs suffer large compute and memory costs because visual token count grows quadratically with image resolution; existing token compression methods are token-centric and lack high-level semantic understanding, leading to erroneous merges, redundancy, or context loss.

Approach

Use an internal segmentation head (Mask2Former on a multi-scale ConvNeXt-L encoder) to produce object masks, merge visual tokens per object (soft or hard masks), and order the resulting object tokens by mask centroid, so the number of tokens adapts to the number of objects in the image.

Key results

• At 640 tokens CORE scores 86.9 on POPE and 1521.6 on MME, matching or exceeding the compared fixed-rate methods on most benchmarks.
• Under adaptive compression CORE (hard mask) averages 63.1 tokens on POPE and reaches 85.6 POPE, versus 86.8 for the full 2880-token LLaVA-NeXT-7B.
• With 2.2% of visual tokens (about 63 on average) CORE retains 97.4% of baseline performance, reducing FLOPs by 16.0x and KV cache by 182.3x.

Surprising findings

• Some compressed CORE settings slightly outperform the full-token CORE baseline, which the authors attribute to a regularization effect of object-centric merging.
• Reversing to a small-object-first merging order causes only a minimal average performance drop.

Limitations

• Memory-bandwidth bottleneck in the vision module limits practical inference speed gains despite lower FLOPs.
• Segmentation head only recognizes 133 predefined categories; OOD objects are partitioned into parts and preserve more tokens.

Experimental setup

Datasets: LLaVA 558K LLaVA-NeXT data

Benchmarks: POPE MME MMBench-CN ScienceQA-IMG SEED-Bench-IMG MMMU

Metrics: accuracy score FLOPs KV cache GPU memory total inference time

Baselines: ToMe FastV PDrop SparseVLM PruMerge+ TRIM VisionZip DART DivPrune SEED-LLaMA Slot-MLLM

Builds on: LLaVA-NeXT Mask2Former ConvNeXt OMG-Seg ToMe

Closest papers by meaning

• PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Models (ICLR 2026 · similarity 0.91)
• Hybrid Token Compression for Vision-Language Models (CVPR 2026 · similarity 0.91)
• ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.91)
• Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity (ICLR 2026 · similarity 0.91)
• DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference (CVPR 2026 · similarity 0.91)
• Vision-centric Token Compression in Large Language Model (NeurIPS 2025 · similarity 0.91)