Research map / 3D vision / 3D generation
Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scene
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Transfer Your Perspective (TYP), a two-stage conditioned latent diffusion model over BEV LiDAR occupancy that generates reference-viewpoint point clouds from ego-car data, enabling collaborative perception training from single-agent datasets like Waymo.
Problem
Collaborative autonomous driving needs multi-agent LiDAR data that is costly to collect; the paper asks whether realistic point clouds from alternative viewpoints can be generated from a real single-agent (ego) scene to turn single-agent datasets into collaborative ones.
Approach
A latent diffusion model over VQ-VAE-encoded BEV occupancy is first trained on object-location conditions using real single-agent data, then a T2I-Adapter-style control module conditions it on the translated ego point cloud using simulated CAV data, with a GAN-style discriminator aligning simulated and real latent features.
Key results
- On OPV2V, Late Fusion trained on TYP-generated data reaches 0.50 AP@0.5 overall vs 0.55 with ground-truth oracle, and AttFuse reaches 0.72 vs 0.71 oracle.
- On V2V4Real (no real CAV data used for generation), Late Fusion trained with TYP-generated data reaches 0.63 AP overall vs 0.61 oracle.
- Pre-training Early Fusion on ColWaymo improves V2V4Real AP from 0.30 (scratch, 0 scenes fine-tune) to 0.35, and from 0.67 to 0.65 at 32-scene fine-tuning comparable, with ColWaymo outperforming OPV2V pre-training in most settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate reference point clouds with TYP rather than use ground-truth reference point clouds | Collaborative perception on OPV2V · OPV2V | AP 0.5 all: 0.49 -> 0.46; 0.55 -> 0.50; 0.77 -> 0.72; 0.71 -> 0.65 | Mixed |
| Generate reference point clouds with TYP rather than use ground-truth reference point clouds | Collaborative perception on V2V4Real · V2V4Real | AP 0.5 all: 0.60 -> 0.53; 0.67 -> 0.63; 0.61 -> 0.56; 0.61 -> 0.48 | Mixed |
| Pre-train collaborative perception models on ColWaymo | Collaborative perception models fine-tuned on V2V4Real · V2V4Real fine-tuning | AP 0.5 all | Helped |
| Add discriminator-based domain adaptation (ablation) | TYP target-domain generation · ColWaymo target-domain generation | JSD: 0.26 -> 0.16 | Helped |
| Add discriminator-based domain adaptation (ablation) | TYP target-domain generation · ColWaymo target-domain generation | MMD: 4.61 × 10−4 -> 1.17 ×10−4 | Helped |
| Use two training stages instead of one (ablation) | TYP generation · OPV2V generation | JSD: 0.11 -> 0.10 | Helped |
| Use two training stages instead of one (ablation) | TYP generation · OPV2V generation | MMD: 6.45 × 10−5 -> 5.28 × 10−5 | Helped |
| Remove dummy ego from adaptation (ablation) | TYP target-domain generation · ColWaymo target-domain generation | MMD: 1.10 ×10−4 -> 1.30 ×10−4 | Hurt |
| Remove self-training and dummy ego from adaptation (ablation) | TYP target-domain generation · ColWaymo target-domain generation | MMD: 1.10 ×10−4; 1.17 ×10−4 | No clear effect |
Surprising findings
- Generated point clouds trained on OPV2V-only generator yield collaborative-perception results on real V2V4Real data that match or slightly exceed the ground-truth oracle for some fusion methods (e.g., late fusion 0.63 vs 0.61).
Limitations
- Focuses on vehicle-like objects; does not generate static entities such as traffic signs or signals.
- Evaluation is offline only, not closed-loop.
- Generation uses binary BEV occupancy with only x-y object locations as semantic condition (no category information).
Experimental setup
Datasets: OPV2V V2V4Real Waymo Open Dataset
Benchmarks: OPV2V V2V4Real
Metrics: AP@0.5 (short, medium, long, all distance ranges) JSD MMD
Baselines: No Fusion (ego only) Early Fusion Late Fusion AttFuse V2X-ViT OPV2V pre-training
Builds on: Latent Diffusion Model T2I-Adapter VQ-VAE Pointpillars
Closest papers by meaning
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.88)
- Privacy-Preserving V2X Collaborative Perception Integrating Unknown Collaborators (AAAI 2025 · similarity 0.88)
- VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.87)
- STAMP: Scalable Task- And Model-agnostic Collaborative Perception (ICLR 2025 · similarity 0.87)
- AGC-Drive: A Large-Scale Dataset for Real-World Aerial-Ground Collaboration in Driving Scenarios (NeurIPS 2025 · similarity 0.86)
- FreeVS: Generative View Synthesis on Free Driving Trajectory (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.