Research map / Vision-language and multimodal / Multimodal LLMs

Towards Unified Human Perception and Machine Understanding: Token Flow Guided Compression Framework

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TFGC, a 1D-token image compression framework that propagates masked tokens via conditional Gaussian prediction (token flow propagation) and aligns decoded tokens to an LVLM's semantic space (token semantic guidance) so LVLMs can consume compressed tokens without image reconstruction.

Problem

Learned image compression is optimized for pixel fidelity with rigid bitrate points, and LVLM-oriented compression relies on reconstructed images, causing semantic degradation at ultra-low bitrates.

Approach

Exploit the holistic, globally propagating nature of 1D token sequences ("token flow") by predicting dropped tokens as samples from a learned conditional Gaussian conditioned on retained tokens, and add a lightweight TSG module trained with a two-stage progressive semantic alignment so tokens feed the LLM directly.

Key results

• At 0.06 bpp, TFGC reaches 49.94 ROUGE-L on MSCOCO vs 39.80 for the best prior compared method (ELIC 39.80, FlexTok 39.10 on the same row set), and 61.49 Acc@0.5 on RefCOCO.
• At 0.02 bpp (0.023 bpp), TFGC achieves 48.32 ROUGE-L on MSCOCO, 54.96 Acc@0.5 on RefCOCO and 62.37 VQAv2 accuracy, beating TiTok-bl128, DiffEIC and FlexTok.
• At 0.06 bpp, TFGC attains LPIPS 0.12 and DISTS 0.11 on ImageNet, the lowest among learned compression methods, though VVC has higher PSNR/SSIM.

Surprising findings

• VVC achieves higher PSNR and SSIM than TFGC yet much worse perceptual metrics, showing numerical fidelity does not track human preference.

Limitations

• Grounding and VQA at 0.06 bpp only match, not exceed, the best single-rate model.
• Reconstruction perceptual quality trails VVC on PSNR/SSIM and the approach relies on a frozen TiTok codebook and frozen LLM.

Experimental setup

Datasets: ImageNet-1K MSCOCO RefCOCO RefCOCOg VQAv2 OKVQA

Benchmarks: MSCOCO Flickr30k RefCOCO RefCOCOg VQAv2 OKVQA ImageNet validation

Metrics: ROUGE-L Acc@0.5 Accuracy PSNR SSIM LPIPS DISTS CLIP-IQA

Baselines: VVC TiTok DiffEIC ELIC FlexTok

Builds on: TiTok FlexTok InternVL2

Closest papers by meaning

• UniCompress: Token Compression for Unified Vision-Language Understanding and Generation (CVPR 2026 · similarity 0.90)
• Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity (ICLR 2026 · similarity 0.90)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.90)
• One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.89)
• ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.89)
• PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models (CVPR 2025 · similarity 0.89)