Research map / Vision-language and multimodal / Visual question answering
Zero-shot 3D Question Answering via Voxel-based Dynamic Token Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dynamic Token Compression (DTC), a training-free voxel-based method that merges visual tokens from multi-view RGB-D images via iterative bipartite matching across growing voxel sizes, cutting visual tokens by over 90% for LLaVA-OneVision while staying near base performance on OpenEQA and ScanQA.
Problem
Multi-frame VLMs used for 3D question answering produce too many visual tokens from multi-view scans; existing token reduction (pooling, frame sampling) loses fine 3D details needed for accurate answers.
Approach
Lift patch-level visual tokens into a 3D feature map and iteratively merge tokens within voxels using bipartite soft matching on visual similarity, increasing voxel size each iteration, so small objects keep high resolution while large/redundant regions are compressed.
Key results
- On OpenEQA, DTC at 8% visual tokens scores 49.3 LLM-Match vs. 56.2 for the base LLaVA-OneVision-7B with 12 frames; at 43% tokens it scores 54.3.
- On ScanQA, DTC at 9% of tokens (789) achieves 26.1 EM@1 vs. 27.6 for the base model (8748 tokens) and 18.5 for frame sampling at 8%.
- Inference throughput increases 47.6% using only 23% of original visual tokens on ScanQA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply dynamic token compression at 56% visual-token usage | LLaVA-OneVision-7B · OpenEQA | LLM-Match: 56.2 -> 55.3 | Hurt |
| Apply dynamic token compression at 43% visual-token usage | LLaVA-OneVision-7B · OpenEQA | LLM-Match: 56.2 -> 54.3 | Hurt |
| Apply dynamic token compression at 26% visual-token usage | LLaVA-OneVision-7B · OpenEQA | LLM-Match: 56.2 -> 54.1 | Hurt |
| Apply dynamic token compression at 8% visual-token usage | LLaVA-OneVision-7B · OpenEQA | LLM-Match: 56.2 -> 49.3 | Hurt |
| Apply dynamic token compression at 54% visual-token usage | LLaVA-OneVision-7B · ScanQA validation | EM@1: 27.6 -> 27.8 | Helped |
| Apply dynamic token compression at 23% visual-token usage | LLaVA-OneVision-7B · ScanQA validation | EM@1: 27.6 -> 27.7 | Helped |
| Apply dynamic token compression at 9% visual-token usage | LLaVA-OneVision-7B · ScanQA validation | EM@1: 27.6 -> 26.1 | Hurt |
| Replace dynamic token compression with frame sampling at 8% token usage (ablation) | LLaVA-OneVision-7B · ScanQA | EM@1: 27.6 -> 18.5 | Hurt |
| Replace dynamic token compression with temporal pooling at 8% token usage (ablation) | LLaVA-OneVision-7B · ScanQA | EM@1: 27.6 -> 18.2 | Hurt |
| Replace dynamic token compression with spatial pooling at 9% token usage (ablation) | LLaVA-OneVision-7B · ScanQA | EM@1: 27.6 -> 22.9 | Hurt |
| Average-pool tokens connected by DTC edges instead of keeping one token (ablation) | Dynamic Token Compression · OpenEQA | LLM-Match: 54.1 -> 51.7 | Hurt |
| Use weighted average pooling for connected tokens instead of keeping one (ablation) | Dynamic Token Compression · OpenEQA | LLM-Match: 54.1 -> 52.9 | Hurt |
Surprising findings
- Keeping one token per matched pair outperformed average or weighted-average merging (54.1 vs 51.7 and 52.9 on OpenEQA), suggesting merging distorts semantics.
Limitations
- Performance drops notably at high compression (e.g., -6.9 on OpenEQA at 8% tokens).
- Evaluated only on OpenEQA and ScanQA with a single base VLM.
Experimental setup
Datasets: OpenEQA ScanQA
Benchmarks: OpenEQA ScanQA
Metrics: LLM-Match EM@1 visual token usage throughput
Baselines: LLaVA-OneVision-7B Frame sampling Vanilla Token Compression MovieChat Chat-UniVi 3D-LLM LEO Scene-LLM ScanQA
Builds on: LLaVA-OneVision token merging (ToMe bipartite soft matching) FastV
Closest papers by meaning
- Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA (AAAI 2024 · similarity 0.89)
- 3D Question Answering via only 2D Vision-Language Models (ICML 2025 · similarity 0.89)
- Rethinking Token Reduction for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy (ICLR 2026 · similarity 0.88)
- Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.88)
- Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.