Research map / 3D vision / 3D generation
Evaluating Model Perception of Color Illusions in Photorealistic Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds RCID, a 19,000-image realistic color illusion dataset (ControlNet-generated contrast/stripe illusions plus filtered COCO images) and evaluates open VLMs on whether they perceive illusions like humans, with a mixed-training baseline.
Problem
Prior VLM color-illusion evaluations use few, often memorized internet or simple synthetic images, so it is unclear whether VLMs perceive color illusions in realistic scenes as humans do.
Approach
An automated pipeline that generates realistic illusion images (procedural shapes conditioned through a ControlNet trained on quantized COCO color grids, plus color-filter overlays on COCO), with human-validated illusion labels via Prolific surveys, enabling large-scale evaluation of human-like versus pixel-faithful VLM responses.
Key results
- After fine-tuning on non-illusion images, models reach over 75% accuracy on non-illusion images while accuracy drops significantly on illusion images.
- Mixed training raises the no-illusion rate for pixel-based prompts to 83.9% (contrast) and 84.5% (stripe), versus 35.4% and 30.3% for base LLaVA-1.5 (7B).
- Few-shot prompting with no-illusion examples increases no-illusion responses by roughly 25% for contrast and stripe illusions, while filter illusions are barely affected.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on pixel-value and human-perception question-answer pairs | LLaVA-1.5 (7B) · Contrast Illusion test set | No-illusion rate: 35.4 -> 83.9 | Helped |
| Train on pixel-value and human-perception question-answer pairs | LLaVA-1.5 (7B) · Filter Illusion test set | No-illusion rate: 20.0 -> 68.0 | Helped |
| Train on pixel-value and human-perception question-answer pairs | LLaVA-1.5 (7B) · Stripe Illusion test set | No-illusion rate: 30.3 -> 84.5 | Helped |
| Use no-illusion answers as 3-shot examples (ablation) | LLaVA-7B · Contrast and Stripe Illusions | No-illusion response proportion: approximately 25% | Helped |
| Use human-like answers as 3-shot examples (ablation) | LLaVA-7B · Contrast Illusion | Human-like response proportion | Hurt |
| Add a chain-of-thought prompt about factors affecting color perception (ablation) | LLaVA-7B · Contrast and Stripe Illusions | No-illusion response proportion | Helped |
| Fine-tune on pixel-value (no-illusion) responses (ablation) | LLaVA-7B · Contrast and Stripe Illusions | No-illusion response proportion: over 90% | Helped |
| Fine-tune on human-like responses (ablation) | LLaVA-7B · Illusion test set | Human-like response proportion | No clear effect |
| Increase contrast between foreground and background colors (ablation) | Humans and LLaVA-1.5 (7B) · Contrast Illusion | Illusion intensity | Hurt |
| Increase stripe image spatial frequency (ablation) | Humans and LLaVA-1.5 (7B) · Stripe Illusion | Illusion strength | Helped |
| Increase model size (ablation) | OFA models · Contrast Illusion | Human-like response proportion | Helped |
| Increase model size (ablation) | OFA models · Contrast Illusion | Pixel-value-consistent response proportion | Hurt |
Surprising findings
- Human-like responses as few-shot examples do not improve human-like behavior and slightly decrease it for contrast illusions.
- Richer color descriptor diversity in training language correlates with lower susceptibility to illusions, while purely vision models (ResNet, ViT) are also deceived by illusions.
Limitations
- Some generated images still look AI-generated, and realism shows no significant correlation with accuracy.
- Filter illusion images are a small subset (2,000) and the test set relies on human-validated labels from a limited Prolific pool.
- Closed-source models (GPT-4o, Gemini) are only evaluated in the appendix.
Experimental setup
Datasets: MS COCO 2017 RCID
Benchmarks: RCID
Metrics: human-like rate no-illusion rate accuracy on non-illusion images deception rate
Baselines: LLaVA-1.5 (7B) InstructBLIP CogVLM InternVL2 Qwen-VL2 OFA Unified-IO GPT-4o Gemini
Builds on: ControlNet GVIL Chain of Thought LLaVA
Closest papers by meaning
- The Art of Deception: Color Visual Illusions and Diffusion Models (CVPR 2025 · similarity 0.82)
- Visual Anagrams: Generating Multi-View Optical Illusions with Diffusion Models (CVPR 2024 · similarity 0.81)
- Towards a Perceptual Evaluation Framework for Lighting Estimation (CVPR 2024 · similarity 0.81)
- Be More Specific: Evaluating Object-centric Realism in Synthetic Images (CVPR 2025 · similarity 0.80)
- GCC: Generative Color Constancy via Diffusing a Color Checker (CVPR 2025 · similarity 0.80)
- 3D Visual Illusion Depth Estimation (NeurIPS 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.