Research map / Vision-language and multimodal / Multimodal LLMs

LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LiteVGGT, which accelerates VGGT on long image sequences via geometry-aware cached token merging (edge/variance-based token partitioning, merge indices cached across layers), plus fine-tuning and FP8 quantization, reaching ~10x speedup on 1000-image scenes.

Problem

VGGT's frame-global attention scales quadratically in time and memory with the number of images, making reconstruction of long sequences (hundreds to 1000 images) slow and out-of-memory.

Approach

Generic token merging ignores that VGGT tokens map one-to-one to image patches and 3D geometry. The method scores tokens by a geometry-aware map (Sobel gradients plus token variance), keeps the top 10% as protected tokens, merges the remaining src tokens into spatially spread dst anchors, and caches merge indices every 6 layers, using stable inter-layer token similarity.

Key results

• On 1000 ScanNet-50 images, LiteVGGT reaches CD 0.428 in 127.2s versus VGGT* at 0.485 and 1275.1s, while vanilla VGGT runs out of memory.
• On DTU, LiteVGGT reconstruction overall error is 0.716 versus FastVGGT 0.739 and VGGT 0.534; CO3Dv2 AUC@30 is 83.2 versus VGGT 86.3.
• Ablation: GA token merging plus fine-tuning, index caching, and FP8 together cut latency from 1275.6s to 127.9s and memory from OOM to 45.31 GiB on 1000 images.

Surprising findings

• Pure edge maps fed to VGGT and Depth Anything V2 still yield coherent geometric reconstructions, suggesting reliance on structural contours.
• Caching merge indices across too many layers (computing only at the first layer) degrades accuracy sharply on DTU, while the 6-layer interval keeps quality nearly unchanged.

Limitations

• Pose accuracy and reconstruction on DTU and CO3Dv2 remain below vanilla VGGT, and LiteVGGT produces slightly less detailed geometry than VGGT.
• Fine-tuning requires about 3 days on 8 H20 GPUs, and the training set is large and mixed.
• Reported hyperparameters such as the GA-map weights alpha/beta and the training data sizes are not specified in the text.

Experimental setup

Datasets: Co3Dv2 BlendMVS DL3DV MegaDepth WildRGB ScanNet++ HyperSim Mapillary Replica MVS-Synth Virtual KITTI ScanNet-50

Benchmarks: ScanNet-50 7Scenes NRGBD Tanks & Temples DTU CO3Dv2

Metrics: Chamfer Distance Accuracy Completeness Normal Consistency F1 AUC latency memory

Baselines: VGGT FastVGGT Fast3R CUT3R StreamVGGT QuantVGGT

Builds on: VGGT ToMe ToMeSD DINOv2

Closest papers by meaning

• VGGT: Visual Geometry Grounded Transformer (CVPR 2025 · similarity 0.89)
• FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry (NeurIPS 2026 · similarity 0.89)
• RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer (NeurIPS 2026 · similarity 0.89)
• TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention (NeurIPS 2026 · similarity 0.88)
• GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction (NeurIPS 2026 · similarity 0.88)
• Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers (NeurIPS 2026 · similarity 0.88)