Research map / Architectures and efficiency / Attention and transformers

GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: KV cache eviction and compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GHOST, a training-free online KV-cache token eviction method for streaming 3D reconstruction that scores tokens using the model's own depth, point-confidence and camera-pose outputs, plus special-token boosting and cosine-similarity layer budgets.

Problem

Streaming 3D reconstruction from long monocular videos needs a KV cache that grows linearly with sequence length; existing eviction relies on attention or key-similarity heuristics that ignore 3D geometric value.

Approach

Hierarchical dual-level importance (frame-level pose change, depth-gradient variance and recency plus token-level feature saliency and depth/point confidence), a deterministic boost for camera and register tokens, and layer-wise budgets derived from offline cosine similarity of layer inputs and outputs.

Key results

• On 7-Scenes at 300 frames, GHOST reduces mean accuracy error from 0.040 (InfiniteVGGT) to 0.023.
• At half budget (600k), GHOST uses 3.51 GB KV cache vs 6.94 GB (49% reduction) with 1.75x speedup and Acc 0.041 vs 0.040.
• On Long3D Library (4,726 frames), GHOST reduces accuracy mean from 1.121 to 0.745 relative to InfiniteVGGT.

Surprising findings

• Key-similarity scores correlate only weakly with camera pose change (Spearman -0.07) and depth gradient variance (-0.31), suggesting attention-based eviction ignores geometric value.

Limitations

• Assumes a static scene; dynamic objects may cause over-retention of frames.
• Not applicable to fully offline joint-attention models such as VGGT.
• Hyperparameters (importance weights) tuned by grid search on held-out sequences.

Experimental setup

Datasets: 7-Scenes NRGBD Bonn Long3D

Benchmarks: 7-Scenes NRGBD Bonn Long3D

Metrics: Accuracy Completeness Normal Consistency Abs Rel delta<1.25 KV cache size speedup

Baselines: VGGT StreamVGGT CUT3R Point3R TTT3R InfiniteVGGT

Builds on: StreamVGGT InfiniteVGGT PyramidKV

Closest papers by meaning

• FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry (NeurIPS 2026 · similarity 0.90)
• LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging (CVPR 2026 · similarity 0.88)
• Rethinking the State Update Gate for Long-Sequence Recurrent 3D Reconstruction (NeurIPS 2026 · similarity 0.87)
• IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer (NeurIPS 2026 · similarity 0.86)
• Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory (NeurIPS 2025 · similarity 0.85)
• Motion Matters: Compact Gaussian Streaming for Free-Viewpoint Video Reconstruction (NeurIPS 2025 · similarity 0.85)