Research map / Video / Video-language understanding
PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PVC, which treats images as static videos and uses causal temporal attention with AdaLN plus an adaptive PixelShuffle compressor to progressively encode new visual information, enabling a unified token compression for images and videos at 64 tokens per frame.
Problem
Existing VLMs compress image and video tokens with different strategies, which limits combined image-video capability; long token sequences from video are costly.
Approach
Standardize all visual inputs as videos (images repeated 4 times as static videos) and progressively encode each frame with causal temporal attention and AdaLN conditioned on timestep, followed by an AdaLN-conditioned PixelShuffle compressor with 16x compression, so later frames only add information missing from earlier ones.
Key results
- PVCInternVL2-8B reaches 73.8 on MVBench, versus 69.1 for the best open-source model within 8B (IXC-2.5), using 64 tokens per frame.
- PVCInternVL2-8B scores 64.1/69.7 on VideoMME (without/with subtitles) and 72.4 on MLVU, improving over InternVL2-8B (54.0/56.9 and 52.0).
- On image benchmarks, the PVC architecture on 2B InternVL2 matches or improves over the baseline (e.g., DocVQA 87.8 vs 86.9, InfoVQA 59.6 vs 58.9), and repeating images adds only about 6% FLOPs for the 8B model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PVC architecture to InternVL2 | InternVL2-8B · MVBench | accuracy: 66.4 -> 73.8 | Helped |
| Add PVC architecture to InternVL2 | InternVL2-8B · VideoMME | accuracy: 54.0/56.9 -> 64.1/69.7 | Helped |
| Add PVC architecture to InternVL2 | InternVL2-8B · DocVQA | accuracy: 91.6 -> 92.5 | Helped |
| Standardize inputs as video by repeating images, without progressive encoding (ablation) | PVCInternVL2-2B · DocVQA | accuracy: 86.4 -> 82.7 | Hurt |
| Add progressive encoding with causal temporal attention (ablation) | PVCInternVL2-2B · MVBench | accuracy: 58.0 -> 61.8 | Helped |
| Add progressive encoding with causal temporal attention (ablation) | PVCInternVL2-2B · MLVU | accuracy: 53.0 -> 54.2 | Helped |
| Add adaptive compression (ablation) | PVCInternVL2-2B · MLVU | accuracy: 54.2 -> 55.1 | Helped |
| Condition AdaLN on temporal embedding instead of no condition (ablation) | PVCInternVL2-2B · MVBench | accuracy: 59.2 -> 62.0 | Helped |
| Condition AdaLN on feature x plus temporal embedding instead of temporal embedding only (ablation) | PVCInternVL2-2B · InfoVQA | accuracy: 57.8 -> 58.5 | Helped |
| Increase image repetitions from 1 to 4 (ablation) | PVC · InfoVQA | accuracy: 43.1 -> 58.5 | Helped |
| Increase image repetitions from 1 to 4 (ablation) | PVC · MMB | accuracy: 72.8 -> 74.9 | No clear effect |
| Use PVC image repetition and progressive encoding instead of InternVL2 | InternVL2-8B · FLOPs | FLOPs: +6.0% | Hurt |
Surprising findings
- Directly standardizing images as static videos without progressive encoding significantly hurts OCR-heavy tasks (DocVQA -3.7, InfoVQA -6.5), showing repeated tokens carry no extra information.
Limitations
- Underperforms some baselines on Egoschema and PerceptTest, possibly due to lack of domain-specific training data.
- Lags behind Qwen2-VL-7B on several OCR-related image benchmarks (e.g., OCRBench -59, TextVQA -4.3).
Experimental setup
Datasets: LAION COYO InternVid WebVid Vript LLaVA-Video InternVL2 training data
Benchmarks: MVBench VideoMME MLVU LongVideoBench NextQA Egoschema PerceptTest ActivityNet-QA AI2D ChartQA DocVQA InfoVQA
Metrics: accuracy
Baselines: InternVL2 Qwen2-VL LLaVA-OneVision LLaVA-Video LongVU Video-CCAM IXC-2.5 Oryx MLLM
Builds on: InternVL2 PixelShuffle AdaLN Video Swin Transformer
Closest papers by meaning
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.91)
- ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.90)
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.90)
- DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models (CVPR 2025 · similarity 0.90)
- Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity (ICLR 2026 · similarity 0.89)
- DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.