Research map / Detection and segmentation / 3D detection and driving
A Black-Box Evaluation Framework for Semantic Robustness in Bird’s Eye View Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a black-box, query-based adversarial evaluation framework that optimises geometric, colour and motion-blur semantic perturbations against camera-based multi-view BEV detectors using a distance-based surrogate of mAP and a simplified DIRECT optimiser (SimpleDIRECT), and benchmarks ten BEV models on nuScenes.
Problem
Random natural corruptions only reveal average-case robustness of camera-based BEV detection models; the worst-case robustness under semantic perturbations is not characterised, and mAP is discontinuous and unsuitable as an optimisation objective.
Approach
A continuous distance-based surrogate of the mAP matching criterion, optimised by SimpleDIRECT, a DIRECT variant that ranks potentially optimal nodes by estimated improvement (value plus diameter times slope) and keeps only the top R nodes, reducing redundant node evaluations.
Key results
- SimpleDIRECT (R=3) matches or beats DIRECT and BO-EI on most perturbations, e.g. reducing matches to 14.4 vs DIRECT 13.8 on PETR R50 geometric perturbation and 7.0 vs 7.0 on motion blur for PETR VoVNet, with lower runtime than BO-EI (9.6 vs 55.1 min on PETR R50 colour).
- In the ten-model benchmark, PolarFormer (VoV) is most robust (e.g. colour mAP 0.329, NDS 0.409) while BEVDet drops to zero NDS and mAP under colour, geometric and motion-blur perturbations.
- On the full validation set, optimised perturbations on PolarFormer VoV reduce mAP to 0.019 (colour) versus 0.460 under random perturbation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SimpleDIRECT instead of random or fixed natural perturbations | BEV robustness evaluation · nuScenes mini-validation, PETR ResNet50 | # matches: Random: 24.4; Natural+: 28.8; Natural-: 23.4; SimpleDIRECT R=3: 23.2 | Helped |
| Use SimpleDIRECT instead of DIRECT | PETR ResNet50 · nuScenes mini-validation, scale and translation | # matches: 13.8 -> 14.4 | Helped |
| Use SimpleDIRECT instead of BO-EI | PETR ResNet50 · nuScenes mini-validation, colour shift | # matches: 19.8 -> 23.2 | Hurt |
| Set the PO-node cap to R=3 rather than R=1 or R=4 (ablation) | SimpleDIRECT on PETR models · nuScenes mini-validation, colour, scale and translation, and motion blur | # matches: ResNet50 colour: 23.2 (R=1), 23.2 (R=2), 23.2 (R=3), 23.2 (R=4); VoVNet 800 colour: 27.4 (R=1), 26.2 (R=2), 26.0 (R=3), | Mixed |
| Add temporal information (ablation) | BEVFormer-small · nuScenes mini-validation, clean input | NDS: 0.279 -> 0.399 | Helped |
| Add temporal information (ablation) | BEVFormer-small · nuScenes mini-validation, colour shift | NDS: 0.223 -> 0.291 | Helped |
| Add temporal information (ablation) | BEVFormer-base · nuScenes mini-validation, colour shift | NDS: 0.261 -> 0.223 | Hurt |
| Optimise perturbations with SimpleDIRECT rather than random search | BEVFormer-small without temporal information · nuScenes full validation, colour shift | mAP: 0.122 -> 0.007 | Hurt |
| Optimise perturbations with SimpleDIRECT rather than random search | PolarFormer VoV · nuScenes full validation, colour shift | mAP: 0.460 -> 0.019 | Hurt |
| Add classification-score minimisation to the objective (ablation) | SimpleDIRECT semantic attack · PETR ResNet and VoVNet, eight reported settings | attack performance | No clear effect |
| Use the distance-based objective rather than the objective that also minimises classification score (ablation) | SimpleDIRECT on PETR ResNet · Colour perturbation | # matches: 23.2 vs 23 | No clear effect |
| Use SimpleDIRECT instead of DIRECT | ImageNet ResNet50 geometric transformation attack · ImageNet classifiers | ASR: 39.29% -> 39.54% | Helped |
Surprising findings
- BEVDet, which shows resilience to random natural corruption, is fully compromised (precision reduced to zero) by optimised semantic perturbations.
- Higher clean-frame precision does not always translate into better robustness under optimised perturbations.
Limitations
- Evaluation limited to three semantic perturbation types and due to GPU limits mainly on the nuScenes mini-validation set.
- Full validation case study perturbs only selected frames and reuses perturbations on other frames.
- BEV models can only be queried sequentially, making optimisation slow.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: mAP NDS number of matches distance metric attack success rate run time
Baselines: Random search Natural+ Natural- DIRECT BO-EI
Builds on: DIRECT NDS Kornia RoboBEV
Closest papers by meaning
- Improving Bird's Eye View Semantic Segmentation by Task Decomposition (CVPR 2024 · similarity 0.89)
- M-BEV: Masked BEV Perception for Robust Autonomous Driving (AAAI 2024 · similarity 0.89)
- Predictive Uncertainty Quantification for Bird's Eye View Segmentation: A Benchmark and Novel Loss Function (ICLR 2025 · similarity 0.88)
- Toward Real-world BEV Perception: Depth Uncertainty Estimation via Gaussian Splatting (CVPR 2025 · similarity 0.88)
- MamBEV: Enabling State Space Models to Learn Birds-Eye-View Representations (ICLR 2025 · similarity 0.86)
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.