Research map / Detection and segmentation / 3D detection and driving
ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZOPP, a zero-shot, human-label-free offboard panoptic perception pipeline for AD scenes that fuses multi-view SAM-Track, LiDAR-camera alignment with parallax filtering, point completion, and neural rendering to produce 2D/3D segmentation, detection, and 4D occupancy on Waymo.
Problem
Offboard auto-labeling for autonomous driving relies on human labels and closed-set taxonomies, struggles with sparse distant objects and open-set categories, and lacks a unified framework covering segmentation, detection, tracking, and occupancy.
Approach
Combine zero-shot vision foundation models (Grounding-DINO, SAM, DeAOT via SAM-Track) extended to multi-view tracking with LiDAR-to-image projection, a parallax occlusion filter that removes background points projected onto foreground masks, a learned point completion network, and neural-rendering SDF reconstruction to decode 4D occupancy flow.
Key results
- ZOPP achieves 37.56 L1 AP overall for vehicle within camera FOV versus 73-89 for fully supervised detectors, with 29.89 AP at 50+m versus 51.24 for CenterPoint at that range.
- Parallax occlusion and noise filtering raises ZOPP vehicle semantic segmentation from 51.6 to 54.2 and point completion raises vehicle box recall from 22.6 to 48.8.
- ZOPP reaches 25.13 occupancy mIoU, below BEVFormer-Fusion (39.05) and LiDAR-only (29.74), but outperforms LiDAR-only on traffic light, sign and cone categories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add point completion (ablation) | ZOPP box interpretation · Waymo Open Dataset val, Vehicle | Recall (IoU): 22.6 -> 48.8 | Helped |
| Add point completion (ablation) | ZOPP box interpretation · Waymo Open Dataset val, Pedestrian | Recall (IoU): 34.2 -> 46.7 | Helped |
| Add point completion (ablation) | ZOPP box interpretation · Waymo Open Dataset val, Cyclist | Recall (IoU): 11.5 -> 22.9 | Helped |
| Add parallax occlusion and noise filtering (ablation) | ZOPP 3D semantic segmentation · Waymo Open Dataset val, Vehicle | class score: 51.6 -> 54.2 | Helped |
| Add parallax occlusion and noise filtering (ablation) | ZOPP 3D semantic segmentation · Waymo Open Dataset val, traffic light | class score: 34.0 -> 34.2 | No clear effect |
| Use mask tracking to leverage temporal context | ZOPP 3D detection · Waymo Open Dataset val, occluded Vehicle | L1 AP (IoU) decrease relative to overall: 11.02% | Helped |
Surprising findings
- Zero-shot ZOPP's 3D occupancy mIoU (25.13) is below the LiDAR-only supervised baseline (29.74) overall, though it wins on some rare categories.
- Its vehicle AP at 50+m is reported as higher than the 0-30m drop trend of baselines, suggesting less distance degradation than supervised detectors in relative terms.
Limitations
- Zero-shot recognition fails on similar categories (construction vehicle, truck, trailer) and uncommon categories (tree trunk, lane marker).
- Neural rendering fails under adverse weather, night, and camera overexposure.
- Objects beyond LiDAR range (e.g., elevated traffic lights) cannot receive 3D boxes.
Experimental setup
Datasets: Waymo Open Dataset Occ3D
Benchmarks: Waymo Open Dataset validation Occ3D
Metrics: AP recall mIoU IoU
Baselines: CenterPoint VoxelRCNN PVRCNN DetZero SAM3D P-Transformer UniSeg TPVFormer BEVFormer BEVFormer-Fusion
Builds on: SAM-Track StreetSurf Grounding-DINO DeAOT PointNet
Closest papers by meaning
- OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection (NeurIPS 2025 · similarity 0.87)
- CorrBEV: Multi-View 3D Object Detection by Correlation Learning with Multi-modal Prototypes (CVPR 2025 · similarity 0.86)
- PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentation (CVPR 2024 · similarity 0.85)
- UniDrive: Towards Universal Driving Perception Across Camera Configurations (ICLR 2025 · similarity 0.85)
- Revisiting Few-Shot Object Detection with Vision-Language Models (NeurIPS 2024 · similarity 0.85)
- SOGDet: Semantic-Occupancy Guided Multi-View 3D Object Detection (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.