Research map / Vision-language and multimodal / Cross-modal learning
StreamingFlow: Streaming Occupancy Forecasting with Asynchronous Multi-modal Data Streams via Neural Ordinary Differential Equation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StreamingFlow, a BEV occupancy-flow predictor that fuses asynchronous LiDAR and camera streams and forecasts occupancy at arbitrary future timestamps by embedding neural ODEs into a GRU-based recurrent network.
Problem
Existing occupancy forecasting methods predict only at fixed, uniformly sampled future snapshots and require strictly synchronized multi-sensor data for fusion, limiting latency and flexibility in autonomous driving.
Approach
Integrates neural ODEs with GRU-style recurrent BEV feature propagation (SpatialGRU-ODE): sensor BEV features trigger state updates whenever they arrive, and the learned derivative propagates the BEV state to any desired future time, so one model supports asynchronous fusion and streaming prediction.
Key results
- On nuScenes, StreamingFlow-base reaches 53.9 IoU and 52.7 PQ for 2s future instance occupancy flow, +1.6 VPQ and +2.4 IoU over FusionAD.
- On Lyft L5, StreamingFlow-tiny reaches 56.9 IoU and 55.9 PQ, surpassing a reproduced BEVFusion by +2.3 IoU and +0.2 PQ.
- Zero-shot prediction at 8s on nuScenes gives 32.5 IoU / 29.8 PQ, only 0.9 IoU and 1.8 PQ below a model trained with 8s labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use StreamingFlow-base instead of FusionAD | nuScenes occupancy forecasting · nuScenes validation | IoU: 51.5 -> 53.9 | Helped |
| use StreamingFlow-base instead of FusionAD | nuScenes occupancy forecasting · nuScenes validation | PQ: 51.1 -> 52.7 | Helped |
| use StreamingFlow-tiny instead of BEVFusion-tiny | Lyft L5 occupancy forecasting · Lyft L5 AV validation | IoU: 54.6 -> 56.9 | Helped |
| use StreamingFlow-tiny instead of BEVFusion-tiny | Lyft L5 occupancy forecasting · Lyft L5 AV validation | PQ: 55.7 -> 55.9 | Helped |
| use StreamingFlow for zero-shot 8s prediction instead of training with 8s labels | StreamingFlow · nuScenes validation, 8s | IoU: 33.6 -> 32.5 | Hurt |
| use StreamingFlow for zero-shot 8s prediction instead of training with 8s labels | StreamingFlow · nuScenes validation, 8s | PQ: 31.6 -> 29.8 | Hurt |
| use asynchronous temporal-then-spatial fusion instead of SpatialGRU-ODE (ablation) | nuScenes occupancy forecasting · nuScenes validation, 2s | IoU: 47.8 -> 44.7 | Hurt |
| use asynchronous temporal-then-spatial fusion instead of SpatialGRU-ODE (ablation) | nuScenes occupancy forecasting · nuScenes validation, 2s | PQ: 46.1 -> 42.7 | Hurt |
| use synchronous spatial-then-temporal fusion instead of SpatialGRU-ODE (ablation) | occupancy forecasting · nuScenes validation, 2s | IoU: 47.8 -> 50.2; 56.9 -> 54.6 | Mixed |
| use synchronous spatial-then-temporal fusion instead of SpatialGRU-ODE (ablation) | occupancy forecasting · Lyft L5 validation, 2s | PQ: 55.9 -> 55.7 | Hurt |
| use midpoint instead of Euler ODE solver (ablation) | StreamingFlow · nuScenes validation, 2s | PQ: 42.9 -> 46.1 | Helped |
| use midpoint instead of Euler ODE solver (ablation) | StreamingFlow · Lyft L5 validation, 2s | IoU: 56.9 -> 55.0 | Hurt |
Surprising findings
- Asynchronous fusion via SpatialGRU-ODE underperforms ordinary temporally synchronized fusion on nuScenes (47.8 vs 50.2 IoU), while it wins on Lyft.
- Denser input streams (10Hz LiDAR, 4Hz camera) yield slightly lower accuracy than the training configuration (47.1-47.6 vs 47.8 IoU).
Limitations
- On nuScenes, the proposed asynchronous fusion is slightly worse than synchronized BEVFusion-style fusion.
- Zero-shot long-horizon accuracy degrades gradually as the horizon grows, and prediction at denser intervals slightly reduces accuracy.
Experimental setup
Datasets: nuScenes Lyft L5
Benchmarks: nuScenes Lyft L5 AV
Metrics: IoU VPQ PQ SQ RQ
Baselines: FIERY StretchBEV BEVerse ST-P3 UniAD MotionNet BE-STI FusionAD BEVFusion
Builds on: GRU-ODE-Bayes BEVFusion Lift-splat-shoot FIERY
Closest papers by meaning
- UnO: Unsupervised Occupancy Fields for Perception and Forecasting (CVPR 2024 · similarity 0.88)
- Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving (AAAI 2025 · similarity 0.86)
- OccProphet: Pushing the Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with an Observer-Forecaster-Refiner Framework (ICLR 2025 · similarity 0.86)
- Spatiotemporal Decoupling for Efficient Vision-Based Occupancy Forecasting (CVPR 2025 · similarity 0.86)
- GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction (CVPR 2025 · similarity 0.85)
- SDGOCC: Semantic and Depth-Guided Bird's-Eye View Transformation for 3D Multimodal Occupancy Prediction (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.