Research map / 3D vision / 3D generation
VQ-Map: Bird's-Eye-View Map Layout Estimation in Tokenized Discrete Space via Vector Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VQ-Map, which learns a VQ-VAE codebook of BEV map patches and then trains a deformable-attention token decoder to predict discrete BEV tokens from camera features for BEV map layout estimation.
Problem
Camera-based BEV map layout estimation suffers from occlusion, depth ambiguity and low resolution, and existing methods lack map prior knowledge; the open question is how to align perspective-view features with generative BEV priors.
Approach
Tokenize ground-truth BEV map patches with a VQ-VAE codebook, then recast dense BEV segmentation as sparse token classification from sparse backbone features via a deformable-attention token decoder; predicted tokens are mapped back to BEV embeddings through the frozen codebook and decoder.
Key results
- Achieves 62.2 mean IoU for surround-view BEV map estimation on nuScenes validation, vs. 59.4 for DDP and 56.6 for BEVFusion (IoU threshold 0.5).
- Achieves 47.6 mean IoU for monocular estimation on nuScenes and 73.4 drivable IoU on Argoverse, outperforming GitNet (45.2) and TaDe (68.3).
- Token classification supervision with focal loss (61.8 mean IoU in ablation) outperforms latent-regression supervision with MSE (60.3 and 60.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BEVFusion with VQ-Map | Camera-only surround-view map estimation · nuScenes val | mIoU: 56.6 -> 62.2 | Helped |
| Replace GitNet with VQ-Map | Monocular map estimation · nuScenes val | Mean IoU: 45.2 -> 47.6 | Helped |
| Replace TaDe with VQ-Map | Monocular map estimation · Argoverse val | Drivable IoU: 68.3 -> 73.4 | Helped |
| Increase token-decoder layers from 2 to 8 (ablation) | VQ-Map token decoder · nuScenes val, surround-view | mIoU: 57.4 -> 62.2 | Helped |
| Increase token-decoder dimension from 256 to 512 (ablation) | VQ-Map token decoder · nuScenes val, surround-view | mIoU: 60.1 -> 62.2 | Helped |
| Use sparse rather than dense BEV features for token prediction (ablation) | VQ-Map alignment · nuScenes val, surround-view | mIoU: 56.3 -> 61.8 | Helped |
| Predict dense maps with sparse features rather than dense features (ablation) | End-to-end map estimation · nuScenes val, surround-view | mIoU: 56.4 -> 59.8 | Helped |
| Use codebook indices with focal loss rather than continuous latent regression with MSE (ablation) | VQ-Map PV-BEV alignment · nuScenes val, surround-view | mIoU: 60.3 -> 61.8 | Helped |
| Use codebook indices with focal loss rather than quantized latent regression with MSE (ablation) | VQ-Map PV-BEV alignment · nuScenes val, surround-view | mIoU: 60.1 -> 61.8 | Helped |
| Add patch-level data augmentations to VQ loss (ablation) | VQ-Map discrete representation learning · nuScenes val, surround-view | mIoU: 61.6 -> 61.8 | Helped |
| Replace deformable attention with standard attention (ablation) | VQ-Map token decoder · nuScenes val, surround-view | mIoU: 61.8 -> 56.5 | Hurt |
| Vary codebook size across K=128, 256, and 512 (ablation) | VQ-Map codebook · nuScenes val, surround-view | mIoU: 61.5; 61.8; 61.7 | No clear effect |
Surprising findings
- Sparse backbone features predicting sparse BEV tokens outperform dense BEV features (LSS-based) for the same token prediction target in the ablation.
Limitations
- Cannot handle position-sensitive semantics with small areas due to patch-level tokenization.
- Tokenization may lose fine spatial detail.
- Currently specialized only for BEV map generation rather than multi-task driving.
Experimental setup
Datasets: nuScenes Argoverse
Benchmarks: nuScenes Argoverse
Metrics: IoU mean IoU Chamfer distance normalized IoU MMD ECE
Baselines: OFT LSS CVT M2BEV BEVFusion MapPrior X-Align MetaBEV DDP IPM VED VPN
Builds on: VQ-VAE ImprovedVQGAN Deformable DETR BEVFusion MapPrior
Closest papers by meaning
- Map It Anywhere: Empowering BEV Map Prediction using Large-scale Public Datasets (NeurIPS 2024 · similarity 0.88)
- Improving Bird's Eye View Semantic Segmentation by Task Decomposition (CVPR 2024 · similarity 0.88)
- A Black-Box Evaluation Framework for Semantic Robustness in Bird’s Eye View Detection (AAAI 2025 · similarity 0.85)
- GA-VLN: Geometry-Aware BEV Representation for Efficient Vision-Language Navigation (CVPR 2026 · similarity 0.85)
- BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation (CVPR 2024 · similarity 0.84)
- M-BEV: Masked BEV Perception for Robust Autonomous Driving (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.