Research map / 3D vision / 3D generation
RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RadOcc, a cross-modal teacher-student distillation for camera-based 3D occupancy prediction that aligns differentiable volume-rendered depth and semantic maps (ray-termination KL plus segment-guided affinity distillation with SAM).
Problem
Camera-only 3D occupancy prediction lacks geometric priors, and feature/logit distillation borrowed from BEV perception fails to help (feature distillation even hurts) in occupancy prediction.
Approach
Distill from a LiDAR+camera teacher by applying differentiable volume rendering to both teacher and student voxel predictions, then align rendered ray termination distributions (RDC loss) and SAM-segment-guided semantic affinity matrices plus rendered-semantic KL (RSC loss).
Key results
- On Occ3D validation, RadOcc with Swin-B reaches 46.06 mIoU vs 44.14 for its baseline (+1.9), and the teacher reaches 49.38.
- On the Occ3D challenge with test-time augmentation, RadOcc-T achieves 49.98 mIoU vs 47.74 for Baseline-T, ranking top-4 per the authors.
- On the BEVDet ResNet50 setting, RDC+RSC improves mIoU from 36.10 to 37.98 (+1.88), while feature distillation (Hinton feature) gives 35.89 (-0.21).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RadOcc rendering-assisted distillation | Swin-B dense occupancy baseline · Occ3D validation | mIoU: 44.14 -> 46.06 | Helped |
| Add RadOcc with test-time augmentation | Swin-B dense occupancy baseline · Occ3D challenge | mIoU: 47.74 -> 49.98 | Helped |
| Add rendering-assisted distillation | TPVFormer camera model · nuScenes-lidarseg test | mIoU: 69.4 -> 71.8 | Helped |
| Add rendering-assisted distillation | BEVDet, ResNet50 · Occ3D | mIoU: 36.10 -> 37.98 | Helped |
| Add probability/logit distillation to rendering distillation (ablation) | BEVDet, ResNet50 with RadOcc · Occ3D | mIoU: 37.98 -> 38.53 | Helped |
| Use feature distillation | BEVDet, ResNet50 · Occ3D | mIoU: 36.10 -> 35.89 | Hurt |
| Use probability and feature distillation | BEVDet, ResNet50 · Occ3D | mIoU: 36.10 -> 35.95 | Hurt |
| Use probability distillation | BEVDet, ResNet50 · Occ3D | mIoU: 36.10 -> 37.00 | Helped |
| Replace rendered ray-distribution consistency with direct rendered-depth alignment using Scale-Invariant Logarithmic loss (ablation) | BEVDet, ResNet50 with RadOcc · Occ3D | mIoU: 36.10 -> 35.08 | Hurt |
| Add rendered depth consistency (RDC) only (ablation) | BEVDet, ResNet50 · Occ3D | mIoU: 36.10 -> 36.76 | Helped |
| Add segment-guided affinity distillation (SAD) only (ablation) | BEVDet, ResNet50 · Occ3D | mIoU: 36.10 -> 37.13 | Helped |
| Add KL divergence between rendered semantics to SAD (ablation) | BEVDet, ResNet50 with SAD · Occ3D | mIoU: 37.13 -> 37.42 | Helped |
Surprising findings
- Direct feature-level distillation from the multi-modal teacher causes negative transfer in occupancy prediction (35.89 vs 36.10 baseline).
- Directly aligning rendered depth with a scale-invariant log loss (Model A) reduces mIoU to 35.08, while aligning ray termination distributions improves it.
Limitations
- Teacher must be a stronger multi-modal (LiDAR+camera) model, requiring LiDAR data during training.
- Sparse LiDAR-segmentation results lag some LiDAR-based methods because voxelized single-sweep LiDAR loses geometric detail.
- RadOcc underperforms the baseline on some classes (e.g., bicycle, pedestrian in Occ3D validation table).
Experimental setup
Datasets: nuScenes Occ3D
Benchmarks: Occ3D nuScenes-lidarseg
Metrics: mIoU
Baselines: MonoScene BEVFormer CTF-Occ PanoOcc BEVDet TPVFormer SurroundOcc BEVDistill Hinton knowledge distillation
Builds on: NeRF volume rendering BEVDet TPVFormer BEVDistill Segment Anything
Closest papers by meaning
- ViPOcc: Leveraging Visual Priors from Vision Foundation Models for Single-View 3D Occupancy Prediction (AAAI 2025 · similarity 0.90)
- AdaOcc: Adaptive 3D Occupancy Prediction for Embodied Tasks (NeurIPS 2026 · similarity 0.89)
- Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.89)
- Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Prediction (CVPR 2025 · similarity 0.89)
- OctOcc: High-Resolution 3D Occupancy Prediction with Octree (AAAI 2024 · similarity 0.88)
- LOMA: Language-assisted Semantic Occupancy Network via Triplane Mamba (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.