Research map / 3D vision / Depth and stereo

Not All Tasks Quantize Equally: Fisher-Guided Quantization for Visual Geometry Transformer

NeurIPS 2026 · 3D vision › Depth and stereo · approach: Feed-forward 3D reconstruction · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Fisher-Guided Quantization (FGQ), a PTQ method for VGGT-style feed-forward 3D models that weights FlatQuant's learnable affine calibration loss by diagonal Fisher sensitivities per task, block, and channel.

Problem

Post-training quantization of multi-task feed-forward 3D reconstruction models (VGGT) degrades tasks unequally, especially point map reconstruction at low bit-width, because uniform calibration losses ignore differing per-task, per-block, and per-channel sensitivities.

Approach

Estimate diagonal empirical Fisher information of each task loss at each block's output channels, normalize and sum across tasks, and use the resulting channel weights in a Fisher-weighted block reconstruction loss for learnable affine transformation calibration (on top of FlatQuant).

Key results

• Under W4A4 on VGGT, FGQ reduces 7-Scenes mean completeness error from 0.085 (QuantVGGT) to 0.059 and improves mean accuracy from 0.053 to 0.048.
• On Co3Dv2 W4A4, FGQ achieves AUC@15 of 0.8887 vs 0.8840 for QuantVGGT and 0.8648 for FlatQuant (FP16 0.9462).
• On π^3 W4A4, FGQ raises Co3Dv2 AUC@3 from 0.2686 (FlatQuant) to 0.5032 against FP16 0.5140.

Surprising findings

• Under W4A4 on VGGT, RTN and QuaRot collapse on point map reconstruction (e.g., ETH3D accuracy error 0.944 and 0.917 vs. 0.263 FP16), far worse than FlatQuant-based methods.

Limitations

• Evaluated only on INT4 weight-activation quantization; MXFP4 and other FP4 formats not studied.
• Fixed small calibration protocol (64 Co3Dv2 clips); calibration set construction not optimized.
• Gains on camera pose estimation are modest compared with dense point map tasks.

Experimental setup

Datasets: Co3Dv2 Re10K 7-Scenes DTU ETH3D KITTI

Benchmarks: Co3Dv2 Re10K 7-Scenes ETH3D DTU KITTI

Metrics: AUC@30 AUC@15 AUC@5 AUC@3 Accuracy Completeness Normal consistency AbsRel SqRel RMSE

Baselines: RTN QuaRot FlatQuant QuantVGGT

Builds on: FlatQuant VGGT QuaRot SpinQuant

Closest papers by meaning

• FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation (CVPR 2025 · similarity 0.88)
• LiDAR-PTQ: Post-Training Quantization for Point Cloud 3D Object Detection (ICLR 2024 · similarity 0.87)
• APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers (CVPR 2025 · similarity 0.87)
• VGGT: Visual Geometry Grounded Transformer (CVPR 2025 · similarity 0.87)
• AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.87)
• Fin3R: Fine-tuning Feed-forward 3D Reconstruction Models via Monocular Knowledge Distillation (NeurIPS 2025 · similarity 0.86)