Research map / Video / Video-language understanding
VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VQToken, which vector-quantizes ViT tokens into a small codebook with a 3D token-hash index map and VQ-Attention, compressing video LLM inputs to about 0.07–0.14% of original length with modest accuracy loss.
Problem
Token-based video LLMs produce very long token sequences; existing pruning and merging methods use fixed ratios, disrupt positional embeddings, and cannot reach extreme compression.
Approach
Cluster ViT tokens into a discrete codebook via (adaptive) K-Means, record each grid token's original (f,h,w) cluster index in a 3D hash map to keep positions, and refine centroids with a VQ-Attention block before feeding them to the LLM.
Key results
- Compresses video token sequences to 0.07% (adaptive, about 13 tokens) of original length with 57.72% NextQA-MC accuracy versus 58.38 for the LLaVA-OV baseline.
- Fixed-length variant with 32 tokens reaches 57.46% NextQA-MC accuracy (0.14% tokens), outperforming Token Pruning, ToMe and VidToMe at the same budget.
- On NextQA-MC, ActNet-QA, LongVideoBench and VideoMME the 0.14% model scores 57.4, 46.3, 39.3 and 38.2 versus 57.2, 50.5, 45.8 and 43.5 for LLaVA-OneVision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compress video tokens with VQ-Token | LLaVA-OneVision 0.5B · NextQA-MC | accuracy: 57.2 -> 57.4 | No clear effect |
| Compress video tokens with VQ-Token | LLaVA-OneVision 0.5B · ActNet-QA | accuracy: 50.5 -> 46.3 | Hurt |
| Compress video tokens with VQ-Token | LLaVA-OneVision 0.5B · LongVideoBench | accuracy: 45.8 -> 39.3 | Hurt |
| Compress video tokens with VQ-Token | LLaVA-OneVision 0.5B · VideoMME | accuracy: 43.5 -> 38.2 | Hurt |
| Use adaptive rather than fixed-length token selection | VQ-Token · Adaptive-length token reduction | accuracy: 57.46 -> 57.72 | Helped |
| Add the discrete codebook (ablation) | LLaVA-OV · LLaVA-OV ablation | accuracy: 57.2 -> 35.2 | Hurt |
| Add the token hash function alongside codebook and randomized VQ-Attn (ablation) | LLaVA-OV · LLaVA-OV ablation | accuracy: 35.2 -> 38.9 | Helped |
| Use a learned rather than randomized token hash function (ablation) | VQToken with codebook and VQ-Attn · LLaVA-OV ablation | accuracy: 46.9 -> 57.5 | Helped |
| Use a learned rather than randomized codebook (ablation) | VQToken with hash and VQ-Attn · LLaVA-OV ablation | accuracy: 37.7 -> 57.5 | Helped |
| Add learned VQ-Attn to codebook and hash (ablation) | LLaVA-OV · LLaVA-OV ablation | accuracy: 35.2 -> 57.5 | Helped |
Surprising findings
- Ablation shows randomized codebook or hash function still gives strong drops, while the full VQ-Attention with learned components is needed to recover accuracy.
- The VQ-Attention-enhanced 0.5B model with 0.14% tokens slightly exceeds LLaVA-OneVision on NextQA-MC despite lower scores on other benchmarks.
Limitations
- Accuracy drops on ActivityNet-QA, LongVideoBench and VideoMME relative to LLaVA-OneVision (e.g., 50.5 to 46.3 and 43.5 to 38.2).
- Evaluation limited to the 0.5B model due to compute; 7B comparison is only through reported numbers.
- Inconsistent reported numbers across tables (e.g., token counts 11664 vs 23328, accuracy 57.5 vs 57.46).
Experimental setup
Datasets: LLaVA-Video-178K NextQA ActivityNet-QA VideoMME LongVideoBench MVBench
Benchmarks: NExT-QA ActivityNet-QA LongVideoBench VideoMME MVBench
Metrics: accuracy token count throughput FLOPs run-time TokDense
Baselines: Token Pruning ToMe VidToMe Interpolating DyCoke LLaVA-OneVision LLaVA-OV-SI
Builds on: LLaVA-OneVision ToMe Vid-ToMe K-Means
Closest papers by meaning
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.91)
- DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models (CVPR 2025 · similarity 0.91)
- Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior (NeurIPS 2025 · similarity 0.91)
- Language Model Beats Diffusion - Tokenizer is key to visual generation (ICLR 2024 · similarity 0.89)
- Vision-centric Token Compression in Large Language Model (NeurIPS 2025 · similarity 0.89)
- When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.