Research map / Video / Video segmentation and tracking
Stereo Vision Conversion from Planar Videos Based on Temporal Multiplane Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends single-view multiplane images to the temporal domain (TMPI) so that adjacent frames, fused via a masked optical flow warping module, fill occluded regions for planar-to-stereo and light-field video conversion.
Problem
Generating high-quality stereo or light-field videos from planar videos is hard because occlusion-exposed regions are missing in the current frame, causing blur or holes in DIBR and single-frame MPI methods.
Approach
Extends MPI from still frames to the temporal domain by predicting per-plane RGB-density jointly from two adjacent frames, and uses masked optical flow warping (MOFW) to propagate only the occluded-region features from the neighbor frame, avoiding ghosting from inaccurate flow.
Key results
- On the 3D movie test set, the method reaches LPIPS 0.027, PSNR 36.33, SSIM 0.960, beating ADAMPI (35.44 PSNR) and others.
- On Middlebury view 0, LPIPS 0.032 and PSNR 30.51 versus ADAMPI 29.71 and 3D Photo 27.21.
- In a 16-observer MOS study on planar-clip light-field synthesis, the method scores 3.96 versus ADAMPI 3.33.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TMPI instead of the strongest baseline, ADAMPI | 3D video conversion · 3D movie test set | LPIPS: 0.036 -> 0.027 | Helped |
| use TMPI instead of the strongest baseline, ADAMPI | 3D video conversion · 3D movie test set | PSNR: 35.44 -> 36.33 | Helped |
| use TMPI instead of the strongest baseline, ADAMPI | 3D video conversion · 3D movie test set | SSIM: 0.956 -> 0.960 | Helped |
| use TMPI instead of ADAMPI | light-field synthesis · Middlebury View0 | PSNR: 29.71 -> 30.51 | Helped |
| use TMPI instead of ADAMPI | light-field synthesis · Middlebury View2 | PSNR: 32.22 -> 32.46 | Helped |
| use TMPI instead of ADAMPI | light-field synthesis · Middlebury View3 | PSNR: 28.71 -> 28.75 | Helped |
| add adjacent-frame information (ablation) | TMPI · 3D movie test set | LPIPS: 0.029 -> 0.027 | Helped |
| add adjacent-frame information (ablation) | TMPI · 3D movie test set | PSNR: 36.28 -> 36.33 | Helped |
| add adjacent-frame information (ablation) | TMPI · 3D movie test set | SSIM: 0.954 -> 0.960 | Helped |
| add masked optical flow warping (ablation) | TMPI · 3D movie test set | LPIPS: 0.049 -> 0.027 | Helped |
| add masked optical flow warping (ablation) | TMPI · 3D movie test set | PSNR: 34.72 -> 36.33 | Helped |
| add masked optical flow warping (ablation) | TMPI · 3D movie test set | SSIM: 0.944 -> 0.960 | Helped |
Surprising findings
- Removing adjacent frames only slightly lowers global metrics (PSNR 36.28 vs 36.33), because the gains are confined to small exposed regions.
Limitations
- Inference speed is too slow for real-time use on limited-compute devices.
- Light-field test set simulates sparse horizontal views from Middlebury and lacks real temporal variation.
- Global metric gains are small because improvements are limited to occluded regions.
Experimental setup
Datasets: 3D movie stereo frames (12,388 pairs) Vimeo90K Middlebury stereo dataset Youku2K
Benchmarks: 3D movie test set (3323 five-frame sequences) Middlebury stereo dataset Youku2K planar clips (light field)
Metrics: PSNR SSIM LPIPS paq2piq ava spaq MOS
Baselines: Deep3D DPSNet+DIBR 3D Photo MINE ADAMPI
Builds on: multiplane images MINE ADAMPI RAFT DPT
Closest papers by meaning
- SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix (ICLR 2025 · similarity 0.83)
- StereoWorld: Geometry-Aware Monocular-to-Stereo Video Generation (CVPR 2026 · similarity 0.83)
- Pushing the Boundaries of View Extrapolation With Multiplane Images (CVPR 2019 · similarity 0.83)
- XPaintNet: An eXtreme Lightweight Framework for Stereoscopic Conversion without Inpainting Network (CVPR 2026 · similarity 0.82)
- SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input (CVPR 2025 · similarity 0.82)
- OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation (CVPR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.