Research map / 3D vision / Depth and stereo
3D Visual Illusion Depth Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a 3D visual illusion depth dataset (~3k scenes, 200k images) showing SOTA monocular and stereo depth models are fooled, and proposes a VLM-guided monocular-stereo fusion network that adaptively fuses disparity cues via a flow-matching confidence map.
Problem
Depth estimation models, both monocular and binocular, are misled by 3D visual illusions (flat artworks, screens, mirrors, transparent surfaces) that simulate 3D structure, a failure mode not systematically studied.
Approach
Build a large illusion dataset from web videos, generative videos, and real stereo+LiDAR captures, then fuse stereo disparity with monocular depth using a VLM (Qwen2-VL-7B) embedding that drives a flow-matching (FLUX-based) confidence map to align and blend the two cues.
Key results
- On real-world illusion regions, the proposed method reaches bad2 26.72 and EPE 1.77 versus RAFT-Stereo's 24.32 and 1.62, i.e., comparable to stereo SOTA while monocular models are far worse (e.g., DepthPro bad2 87.08).
- On the Booster dataset, the full VLM-based fusion achieves bad2 56.77 and EPE 7.32 on transparent regions, versus 67.84 and 9.55 for RAFT-Stereo, and 13.84 bad2 overall versus 16.77 for MochaStereo.
- Ablation on Booster shows the VLM-based fusion improves bad2 from 80.38 (stereo only) to 56.77, with the stereo-plus-fixed-monocular variant performing worse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune Marigold on 3D-Visual-Illusion data | Marigold · 3D-Visual-Illusion illusion regions | EPE: 21.16 -> 13.67 | Helped |
| fine-tune Marigold on 3D-Visual-Illusion data | Marigold · 3D-Visual-Illusion illusion regions | bad2: 65.67 -> 74.82 | Hurt |
| fine-tune Marigold on 3D-Visual-Illusion data | Marigold · 3D-Visual-Illusion non-illusion regions | EPE: 7.61 -> 7.10 | Helped |
| fine-tune Marigold on 3D-Visual-Illusion data | Marigold · 3D-Visual-Illusion non-illusion regions | bad2: 49.18 -> 55.63 | Hurt |
| add simple post fusion of monocular depth to the stereo-only model (ablation) | Booster stereo-only baseline · Booster | EPE: 15.11 -> 8.36 | Helped |
| add monocular features to PF (ablation) | PF · Booster | bad2: 69.89 -> 68.46 | Helped |
| add monocular features to PF (ablation) | PF · Booster | EPE: 8.36 -> 9.25 | Hurt |
| use adaptive post fusion instead of PF (ablation) | MF fusion · Booster | EPE: 9.25 -> 9.59 | Hurt |
| use stereo fusion with fixed monocular disparity instead of PF (ablation) | MF fusion · Booster | EPE: 8.36 -> 10.40 | Hurt |
| use VLM-based fusion instead of simple PF (ablation) | Monocular-stereo fusion · Booster | bad2: 69.89 -> 56.77 | Helped |
| fine-tune stereo models on 3D-Visual-Illusion data | RAFT-Stereo and Selective-IGEV · Booster Trans regions | EPE: RAFT-Stereo 9.55 -> 9.24; Selective-IGEV 9.50 -> 9.42 | Mixed |
| fine-tune stereo models on 3D-Visual-Illusion data | RAFT-Stereo and Selective-IGEV · Booster NonTrans regions | EPE: RAFT-Stereo 3.23 -> 7.12; Selective-IGEV 3.60 -> 5.97 | Hurt |
Surprising findings
- Fine-tuning stereo models on the illusion data gives only limited gains on transparent regions and degrades non-transparent regions, attributed to conflicting mirror versus other illusion cues.
- Monocular models like DepthPro perform best on Booster mirror/transparent scenes, while stereo methods are fooled by mirrors and specular objects.
Limitations
- Virtual data pipeline relies on manual segmentation and annotation, which is labor-intensive.
- Real-world subset covers only inpainting, picture, and replay illusions with 617 test frames; VLM-driven fusion is computationally expensive.
Experimental setup
Datasets: SceneFlow 3D-Visual-Illusion Booster Middlebury
Benchmarks: 3D-Visual-Illusion real-world test Booster Middlebury
Metrics: EPE bad2 bad3 bad5 AbsRel RMSE delta1
Baselines: Depth Anything V2 Metric3D DepthPro Marigold Dust3R VGGT RAFT-Stereo Selective-IGEV MochaStereo StereoAnything
Builds on: RAFT-Stereo Selective-IGEV Depth Anything V2 FLUX InstantSplat
Closest papers by meaning
- DepthCues: Evaluating Monocular Depth Perception in Large Vision Models (CVPR 2025 · similarity 0.87)
- Visual Anagrams: Generating Multi-View Optical Illusions with Diffusion Models (CVPR 2024 · similarity 0.84)
- The Art of Deception: Color Visual Illusions and Diffusion Models (CVPR 2025 · similarity 0.84)
- Mono2Stereo: A Benchmark and Empirical Study for Stereo Conversion (CVPR 2025 · similarity 0.84)
- The 3D-PC: a benchmark for visual perspective taking in humans and machines (ICLR 2025 · similarity 0.83)
- Hallo3D: Multi-Modal Hallucination Detection and Mitigation for Consistent 3D Content Generation (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.