Research map / Label-efficient and robust learning / Domain adaptation and generalization
Towards foundational LiDAR world models with efficient latent flow matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies transferability of a LiDAR world model pretrained on nuScenes to beam-count, indoor, and semantic occupancy forecasting, and proposes a Swin-Transformer VAE plus latent conditional flow matching model that is far cheaper than prior occupancy world models.
Problem
Existing LiDAR world models are trained narrowly for single domains and are computationally heavy (high-dimensional latents, thousands of epochs, slow diffusion sampling), and semantic occupancy forecasting depends on costly labels.
Approach
Pretrain a latent conditional flow matching forecaster with a Swin-Transformer VAE (192x compression, 16-dim latent) on unlabeled LiDAR, then fine-tune for new domains with VAE alignment (full VAE fine-tuning or cosine-similarity alignment loss) to get transfer gains with few labels.
Key results
- Pretrained model with full fine-tuning gives up to 11.17% absolute mIoU gain over training from scratch at 10% semantic fine-tuning data and beats from-scratch in 30/36 comparisons.
- Swin-Transformer VAE reaches 92.8% mIoU at 192x compression, versus OccWorld and DOME trailing by over 11% mIoU.
- Semantic occupancy forecasting reaches 33.17% 1s mIoU and 22.22 FPS at 389.46 GFLOPs per frame with 30.37M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed model instead of RenderWorld | nuScenes semantic occupancy forecasting · nuScenes semantic occupancy forecasting, 1s | mIoU: 28.69 -> 33.17 | Helped |
| Use the proposed model instead of OccWorld | nuScenes semantic occupancy forecasting · nuScenes semantic occupancy forecasting, Avg | mIoU: 17.13 -> 23.33 | Helped |
| Use the proposed model instead of DOME | nuScenes semantic occupancy forecasting · nuScenes semantic occupancy forecasting, Avg | IoU: 32.13 -> 31.78 | Hurt |
| Use the proposed model instead of DOME | nuScenes semantic occupancy forecasting · nuScenes semantic occupancy forecasting | GFLOPs per frame: 8891.98 -> 389.46 | Helped |
| Use the proposed model instead of UniScenes | occupancy reconstruction at compression ratio 32x · Occupancy reconstruction, compression ratio 32x | mIoU: 92.1 -> 99.2 | Helped |
| Pretrain VAE and CFM, then fine-tune fully instead of training from scratch | High-beam adaptation · High-beam adaptation, 10% data, 3s | IoU: 10.77 -> 12.33 | Helped |
| Pretrain VAE and CFM, then fine-tune fully instead of training from scratch | Indoor occupancy forecasting · Indoor occupancy forecasting, 10% data, 1s | IoU: 27.28 -> 30.40 | Helped |
| Pretrain VAE and CFM, then fine-tune fully instead of training the VAE from scratch | Semantic occupancy forecasting · Semantic occupancy forecasting, 10% data, 1s | mIoU: 75.81 -> 87.97 | Helped |
| Add U-Net architecture (ablation) | Dynamic learning · Dynamic learning, 3s | mIoU: 17.42 -> 17.77 | Helped |
| Add conditional flow matching (ablation) | Dynamic learning with U-Net · Dynamic learning, 3s | mIoU: 17.77 -> 20.14 | Helped |
| Add 3D convolution after Temporal DiT (ablation) | Dynamic learning with U-Net and CFM · Dynamic learning, 3s | mIoU: 20.14 -> 20.67 | Helped |
| Add classifier-free guidance (ablation) | Dynamic learning with U-Net, CFM, and 3D convolution · Dynamic learning, 3s | mIoU: 20.67 -> 21.05 | Helped |
Surprising findings
- Training from scratch surpasses all fine-tuning methods on indoor forecasting once fine-tuning data exceeds about 25-50%.
- A from-scratch VAE gets higher reconstruction mIoU/IoU than the pretrained VAE at 100% data, yet fine-tuning performance is worse, suggesting latent alignment matters more than reconstruction.
Limitations
- Indoor pretraining domain is distinct from outdoor pretraining, and the authors hypothesize gains would persist only with indoor data in pretraining.
- Pretrained performance drops after more than 50% fine-tuning data on indoor tasks, attributed to limited indoor data diversity.
Experimental setup
Datasets: nuScenes nuScenes semantic occupancy KITTI-360 indoor LiDAR dataset (OSO-128, Clearpath Jackal)
Benchmarks: nuScenes occupancy forecasting KITTI-360 indoor LiDAR occupancy forecasting
Metrics: mIoU IoU FID KID FVD NLL (bits/dim) FPS GFLOPs
Baselines: OccWorld DOME DynamicCity OccLLama RenderWorld UniScenes Occ4D ViDAR
Builds on: Stable Diffusion 3 VAE rectified flow DiT LoRA OccWorld
Closest papers by meaning
- DIO: Decomposable Implicit 4D Occupancy-Flow World Model (CVPR 2025 · similarity 0.90)
- UnO: Unsupervised Occupancy Fields for Perception and Forecasting (CVPR 2024 · similarity 0.87)
- WeatherGen: A Unified Diverse Weather Generator for LiDAR Point Clouds via Spider Mamba Diffusion (CVPR 2025 · similarity 0.87)
- TREND: Unsupervised 3D Representation Learning via Temporal Forecasting for LiDAR Perception (NeurIPS 2025 · similarity 0.86)
- LiT: Unifying LiDAR "Languages" with LiDAR Translator (NeurIPS 2024 · similarity 0.86)
- LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.