Research map / Architectures and efficiency / Attention and transformers

Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Token merging and masked pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Training-free two-stage token selection (diversity-based inter-frame FPS selection plus layer-adaptive intra-frame downsampling/local attention) that restricts key/value tokens in global attention of visual geometry transformers, speeding up VGGT and π³ by over 85% on 500-frame scenes while matching or improving accuracy.

Problem

Global attention in visual geometry transformers (VGGT, π³) scales quadratically with the number of input frames and tokens, making long multi-view sequences prohibitively slow for 3D reconstruction.

Approach

Restrict each query's key/value budget with hierarchical token selection: first pick a diverse set of K anchor frames via farthest point sampling over place-recognition features, then apply layer-dependent intra-frame strategies (local attention in earliest layers, spatial downsampling in diluted-attention layers, full tokens in spiking middle layers), guided by attention entropy analysis.

Key results

• On 7-Scenes camera pose, the σ=2 variant on VGGT achieves ATE 0.0673 vs. VGGT base 0.0698, with no training required.
• Accelerates visual geometry transformers by over 85% on scenes with 500 images while maintaining or improving baseline performance (abstract).
• Diversity-based (FPS) inter-frame selection substantially outperforms co-visibility and attention-based frame selection heuristics at K=25 frames.

Surprising findings

• Sparsifying tokens in global attention can improve over the full-attention base model, suggesting current visual geometry transformers are not optimally trained.
• Uniform intra-frame downsampling across all global attention layers causes noticeable degradation even at sigma=2, whereas early layers tolerate aggressive pruning.

Limitations

• Slightly slower than retrained LiteVGGT despite comparable acceleration
• Entropy-based adaptive layer partitioning and token-level diversity add overhead, so they were not adopted
• Late-layer intra-frame selection is sensitive to the threshold choice

Experimental setup

Datasets: 7-Scenes Neural RGB-D TUM-Dynamics Bonn

Benchmarks: 7-Scenes Neural RGB-D TUM-Dynamics Bonn

Metrics: ATE RPE-rot RPE-trans Accuracy Completion Normal Consistency Abs Rel RMSE Log RMSE Sq Rel

Baselines: FastVGGT SparseVGGT Co-Me LiteVGGT Speed3R AVGGT

Builds on: VGGT AVGGT DUSt3R farthest point sampling keyframe-based SLAM

Closest papers by meaning

• TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention (NeurIPS 2026 · similarity 0.88)
• LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging (CVPR 2026 · similarity 0.88)
• GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers (ICLR 2024 · similarity 0.87)
• How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need? (NeurIPS 2025 · similarity 0.86)
• One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs (CVPR 2026 · similarity 0.86)
• Understanding Multi-View Transformers (NeurIPS 2026 · similarity 0.86)