Research map / 3D vision / 3D generation
WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects Under Occlusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates a large automatically labeled dataset of occluded vehicles and people by 3D-compositing unoccluded time-lapse objects into median backgrounds, and trains Swin/Mask R-CNN and 3D reconstruction models on it.
Problem
Object understanding (2D amodal detection/segmentation/keypoints and 3D pose/shape) fails under severe occlusion in urban scenes, largely because ground-truth annotations for occluded objects are scarce.
Approach
Mine unoccluded objects from stationary time-lapse cameras, use off-the-shelf 2D/3D predictions as pseudo-groundtruth, and composite 3D-posed objects back into a median background ordered by depth (clip-art style) so that occlusion configurations are physically plausible, extending the 2D-only WALT compositing.
Key results
- Model finetuned on WALT3D reaches +13.9 mask AP over WALT2D at 15% of training data, and 25% of WALT3D data matches 60% of WALT2D data.
- Vehicle 2D keypoint PCK@0.1 improves from 66.41% to 80.12% with the new keypoint dataset, improving pseudo-label quality.
- On MOT17-train with SORT, WALT3D-finetuned detector raises HOTA from 40.1 to 42.0 and MOTA from 42.2 to 47.2 versus COCO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on WALT3D instead of WALT2D | Swin detector and segmentation model · KINS | Mask AP: 22.4 -> 23.5 | Helped |
| Fine-tune on WALT3D instead of WALT2D | Swin detector and segmentation model · COCO-A | Mask AP: 36.4 -> 38.7 | Helped |
| Add WALT3D rather than WALT2D training data | Swin model trained with COCO, COCO-A, and KINS · KINS | Mask AP: 33.3 -> 35.6 | Helped |
| Add WALT3D rather than WALT2D training data | Swin model trained with COCO, COCO-A, and KINS · COCO-A | Mask AP: 43.5 -> 45.3 | Helped |
| Fine-tune the detector with WALT3D | SORT with a COCO-pretrained detector · MOT17-train | MOTA: 42.2 -> 47.2 | Helped |
| Use 3D-based WALT3D compositing instead of 2D-based WALT2D | Swin vehicle segmentation model · WALT3D evaluation, 15% of training data | Mask AP: +13.9 APmask | Helped |
| Use the learned Occlusion Classifier rather than the heuristic filter (ablation) | WALT3D data-generation pipeline · Downstream evaluations | AP: less than ±0.5 AP | No clear effect |
| Train with the new vehicle 2D keypoints dataset | Vehicle keypoint localization · PCK@0.1 | PCK@0.1: 66.41% to 80.12% | Helped |
Surprising findings
- The occlusion classifier outperformed the simple IoU/bottom-overlap heuristic but gave less than ±0.5 AP improvement downstream, so it was not kept as a mandatory component.
Limitations
- Generalization across diverse camera views is still limited.
- Requires a mean shape or parametric object model, which is hard for rare object categories.
- Appearance inconsistencies such as lighting variation remain in clip-art images.
Experimental setup
Datasets: WALT time-lapse dataset COCO PASCAL3D+ KITTI-3D CarFusion ApolloCar3D KINS COCO-Amodal MOT17
Benchmarks: KINS COCO-Amodal MOT17-train WALT time-lapse held-out test set
Metrics: box AP mask AP PCK MPJPE HOTA MOTA DetA IDF1
Baselines: SWIN (COCO) WALT2D Occ-Net 3DRCNN HMR 2.0
Builds on: WALT Occ-Net HMR 2.0 Swin Transformer Mask R-CNN
Closest papers by meaning
- DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes (ICLR 2025 · similarity 0.86)
- Robust 3D Shape Reconstruction in Zero-Shot from a Single Image in the Wild (CVPR 2025 · similarity 0.86)
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.85)
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos (CVPR 2025 · similarity 0.85)
- EA3D: Online Open-World 3D Object Extraction from Streaming Videos (NeurIPS 2025 · similarity 0.85)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.