Research map / 3D vision / 3D generation

Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Prediction

CVPR 2025 · 3D vision › 3D generation · approach: Vision-based 3D occupancy prediction · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GDFusion, an RNN-style temporal fusion framework for vision-based 3D semantic occupancy prediction that reinterprets RNN updates as gradient descent and fuses scene-level, motion, geometry, and voxel-level temporal cues.

Problem

Existing temporal fusion in vision-based 3D semantic occupancy prediction (VisionOcc) focuses only on voxel-level feature fusion, and memory grows with stored history frames; other temporal cues (scene, motion, geometry) are unexploited.

Approach

Reinterpret the vanilla RNN update as a gradient descent step minimizing a current-history discrepancy loss; the gradients act as temporal residuals that fuse heterogeneous representations (volumes, network parameters, flow/motion, probabilistic depth) within a single-frame-sized hidden state.

Key results

• GDFusion improves BEVDetOcc, FB-Occ and ALOcc by 1.0-1.9 mIoU on Occ3D with ResNet-50 (e.g., BEVDetOcc-GF 43.6 vs 41.9 SF; ALOcc-GF 46.5 vs 45.5).
• Reduces inference memory by roughly 30%-72% (e.g., BEVDetOcc 10717 MB to 3017 MB); with Swin-Base ALOcc-GF reaches 51.8 mIoU vs 50.6 at 1/3 memory.
• Full ablation model reaches 43.3 mIoU vs 38.0 baseline, with each temporal cue contributing complementary gains.

Measured effects reported in the paper

Surprising findings

• Vanilla RNN-based voxel-level fusion matches SOLOFusion performance while using a single-frame-sized hidden state.
• RNN-based fusion does not need the delayed-fusion training schedule used by prior methods.

Limitations

• Gains on the full Occ3D benchmark are reported only for a few baselines (BEVDetOcc, FB-Occ, ALOcc), with limited evaluation beyond nuScenes.
• Scene-level parameters are updated online per scene, adding training and inference complexity.
• Results depend on the chosen ResNet-50 / Swin-Base backbones and 256x704 or 512x1408 input sizes.

Experimental setup

Datasets: nuScenes Occ3D

Benchmarks: Occ3D (nuScenes)

Metrics: mIoU mIoUD IoU inference memory

Baselines: SOLOFusion BEVDetOcc-SF FB-Occ ALOcc UniOCC SurroundSDF FlashOCC COTR ViewFormer OPUS BEVFormer OSP

Builds on: SOLOFusion BEVDet FB-Occ ALOcc test-time training

Closest papers by meaning

• Exploiting Continuous Motion Clues for Vision-Based Occupancy Prediction (AAAI 2025 · similarity 0.89)
• AdaOcc: Adaptive 3D Occupancy Prediction for Embodied Tasks (NeurIPS 2026 · similarity 0.89)
• Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.89)
• STCOcc: Sparse Spatial-Temporal Cascade Renovation for 3D Occupancy and Scene Flow Prediction (CVPR 2025 · similarity 0.89)
• RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation (AAAI 2024 · similarity 0.89)
• LOMA: Language-assisted Semantic Occupancy Network via Triplane Mamba (AAAI 2025 · similarity 0.88)