Research map / Detection and segmentation / 3D detection and driving
SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SQS, a query-based 3D Gaussian splatting pre-training method for sparse perception models that reconstructs multi-view RGB and depth, with query interaction to transfer pre-trained Gaussian queries into downstream occupancy and detection models.
Problem
Pre-training for autonomous driving has relied on dense BEV or volumetric representations, which do not transfer to sparse query-based perception models (SPMs) that lack explicit spatial structure.
Approach
Predict 3D Gaussians from a small set of learnable Gaussian queries during pre-training and supervise them with self-supervised splatting of multi-view RGB and LiDAR-derived depth; during fine-tuning, a kNN-based local query interaction fuses pre-trained Gaussian queries with task-specific queries.
Key results
- On SurroundOcc val, GaussianFormer + SQS reaches 20.40 mIoU vs 19.10 for GaussianFormer (+1.3 mIoU, +1.69 IoU).
- On nuScenes val with ResNet50 at 704x256, SparseBEV + SQS reaches 56.6 NDS / 45.2 mAP vs 55.8 / 44.8 for SparseBEV.
- With ResNet101 at 1408x512, SparseBEV + SQS reaches 60.2 NDS / 50.9 mAP vs 59.2 / 50.1; with 10% fine-tuning data occupancy mIoU gain is about +3.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SQS pre-training | GaussianFormer · SurroundOcc val | mIoU: 19.10 -> 20.40; +1.30 mIoU | Helped |
| Add SQS pre-training | SparseBEV † (ResNet50, 704 × 256) · nuScenes val | NDS: 55.8 -> 56.6 | Helped |
| Add SQS pre-training | SparseBEV † (ResNet101, 1408 × 512) · nuScenes val | NDS: 59.2 -> 60.2; 1.0 NDS improvements | Helped |
| Use RGB rendering only instead of RGB and depth rendering (ablation) | SQS pre-training · SurroundOcc val, quarter training data | mIoU: 15.2 -> 12.2; reduction of 3.0 | Hurt |
| Use depth rendering only instead of RGB and depth rendering (ablation) | SQS pre-training · SurroundOcc val, quarter training data | mIoU: 15.2 -> 17.3; improvements of 2.1 | Helped |
| Use both RGB and depth rendering instead of RGB rendering only (ablation) | SQS pre-training · SurroundOcc val, quarter training data | mIoU: 12.2 -> 17.5 | Helped |
| Add query interaction to the pre-trained model during fine-tuning (ablation) | SQS · SurroundOcc val, quarter training data | mIoU: 15.2 -> 15.9; 0.7 mIoU | Helped |
| Add query interaction without pre-training (ablation) | Baseline · SurroundOcc val, quarter training data | IoU: 25.8 -> 25.7 | No clear effect |
| Load the pretrained image backbone and FPN neck (ablation) | SQS fine-tuning · SurroundOcc val, quarter training data | mIoU: 18.0 | Helped |
| Fine-tune with only 10% of the data instead of full fine-tuning data | SQS · SurroundOcc occupancy prediction | mIoU: about +3.7 mIoU | Helped |
Surprising findings
- Using RGB rendering alone during pre-training hurts fine-tuning (-2.0 IoU, -3.0 mIoU), while depth rendering alone helps (+2.1 IoU and mIoU).
- Query interaction without pre-training (Model E) gives negligible gain (-0.1 IoU, +0.1 mIoU).
Limitations
- Extra computation and memory cost from the plug-in pre-training model.
- Pre-training queries are insufficiently utilized across different downstream tasks.
- Pre-training does not include semantic information, so queries are not task-distinguishable.
Experimental setup
Datasets: nuScenes SurroundOcc nuImages
Benchmarks: SurroundOcc val nuScenes val
Metrics: mIoU IoU NDS mAP mATE mASE mAOE mAVE mAAE
Baselines: GaussianFormer OccFormer SurroundOcc TPVFormer SparseBEV StreamPETR Sparse4Dv2 BEVFormer
Builds on: 3D Gaussian Splatting SparseBEV GaussianFormer SurroundOcc FCOS3D
Closest papers by meaning
- ODG: Occupancy Prediction Using Dual Gaussians (NeurIPS 2025 · similarity 0.86)
- DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Input (AAAI 2025 · similarity 0.86)
- SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction (CVPR 2024 · similarity 0.86)
- CorrBEV: Multi-View 3D Object Detection by Correlation Learning with Multi-modal Prototypes (CVPR 2025 · similarity 0.85)
- Navigation-Guided Sparse Scene Representation for End-to-End Autonomous Driving (ICLR 2025 · similarity 0.85)
- SparseAlign: a Fully Sparse Framework for Cooperative Object Detection (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.