Research map / Image generation / Autoregressive and latent generation
The Art of Deception: Color Visual Illusions and Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that DDIM inversion in Stable Diffusion produces human-like brightness/color illusion shifts in latent trajectories, and uses this to replicate illusions and generate new illusions in T2I images via latent-optimization with DeepFloyd.
Problem
Understanding how visual illusions (brightness and color) are encoded in diffusion models, and whether diffusion models can both replicate human-like illusions and be used to generate new illusions in natural images.
Approach
Intermediate DDIM inversion latents of diffusion models shift in lightness/color in the same direction as human perception of illusions; this is exploited by measuring target-region intensities at intermediate noisy steps and optimizing latents toward a desired perceived intensity during text-to-image generation.
Key results
- DDIM 10-step inversion reaches 84.17% PAS at threshold 0.8 on BRI3L versus 36.58% for ODOG and 0% for DN-Net/RestoreNet.
- On IllusionVQA, GVIL and HallusionBench, DDIM 10 steps achieves 90.4%, 96.7% and 100% PAS respectively, vs. best baseline CIWAM at 56.52%, 63.63% and 83.33%.
- In a psychophysical study with 15 observers, generated illusions were judged as having different targets in 64% (mean) of cases vs. 13% for control images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DDIM inversion instead of the VAE alone (ablation) | Stable Diffusion 1.4 · BRI3L | PAS: 0.00 -> 84.17 | Helped |
| use DDIM inversion instead of the VAE alone (ablation) | Stable Diffusion 1.4 · IllusionVQA | PAS: 4.70 -> 90.40 | Helped |
| use DDIM inversion instead of the VAE alone (ablation) | Stable Diffusion 1.4 · GVIL | PAS: 21.30 -> 96.70 | Helped |
| use DDIM inversion instead of the VAE alone (ablation) | Stable Diffusion 1.4 · HallusionBench | PAS: 0.00 -> 100.00 | Helped |
| increase inversion steps from 5 to 10 (ablation) | Stable Diffusion 1.4 DDIM · BRI3L (threshold 0.8) | PAS: 44.02 -> 84.17 | Helped |
| increase inversion steps from 5 to 10 (ablation) | Stable Diffusion 1.4 DDIM · BRI3L (threshold 0.9) | PAS: 91.43 -> 98.85 | Helped |
| increase inversion steps from 5 to 10 (ablation) | Stable Diffusion 1.4 DDIM · BRI3L (threshold 1.0) | PAS: 99.88 -> 100.00 | Helped |
| generate images with enforced visual-illusion targets | Text-to-image diffusion model · Generated images, observer study | images judged to have different targets: 13% (mean) -> 64% (mean) | Helped |
Surprising findings
- A plain VAE (no DDIM inversion) yields near-zero PAS on most datasets, indicating the diffusion trajectory rather than the autoencoder drives the illusion alignment.
Limitations
- Replication fails for images close to the natural image distribution because the target regions are spatially destroyed during inversion.
- Generation relies on user-defined target regions at inference, constraining the T2I model's expressiveness.
- Evaluation on VQA-based datasets relies on manually created segmentation masks.
Experimental setup
Datasets: BRI3L IllusionVQA GVIL HallusionBench DiffusionDB
Benchmarks: BRI3L IllusionVQA GVIL HallusionBench
Metrics: Perception Accuracy Score (PAS) mean intensity difference
Baselines: ODOG CIWAM DN-Net RestoreNet VAE Gomez-Villa et al. generation Hirsch et al. generation Roy et al. generation
Builds on: DDIM Stable Diffusion DeepFloyd classifier-free guidance
Closest papers by meaning
- Visual Anagrams: Generating Multi-View Optical Illusions with Diffusion Models (CVPR 2024 · similarity 0.86)
- Understanding Hallucinations in Diffusion Models through Mode Interpolation (NeurIPS 2024 · similarity 0.85)
- Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusions (CVPR 2026 · similarity 0.85)
- Latent Representation Matters: Human-like Sketches in One-shot Drawing Tasks (NeurIPS 2024 · similarity 0.84)
- A Computational Framework for Modeling Emergence of Color Vision in the Human Brain (ICLR 2025 · similarity 0.84)
- 3D Visual Illusion Depth Estimation (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.