Research map / Vision-language and multimodal / Multimodal LLMs
Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IllusionBench, three datasets of abstract shapes hidden in naturalistic scenes via ControlNet/Illusion Diffusion, and shows that zero-shot and in-context VLMs struggle to recognize them while fine-tuned CLIP can learn them with domain generalization.
Problem
Vision-language models are assumed to rely more on shape than earlier classifiers, but it is unclear whether they can recognize abstract shapes represented by arrangements of elements within coherent, naturalistic scenes, unlike existing texture-biased benchmarks such as Cue Conflict and Stylized-Im
Approach
Generate images in which an abstract shape (from classes, logos, or icons) is embedded in a naturalistic scene using ControlNet-conditioned Stable Diffusion (Illusion Diffusion), then evaluate shape and scene recall of VLMs in zero-shot, in-context, and domain-generalization settings.
Key results
- placeholder
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune all CLIP parameters with RegMixUp instead of using zero-shot prediction | CLIP · IllusionBench-IN unseen test domains | accuracy: 73.00\% | Helped |
| fine-tune all CLIP parameters with MixUp instead of using zero-shot prediction | CLIP · IllusionBench-IN unseen test domains | accuracy: 71.79\% | Helped |
| use ERM full-parameter fine-tuning rather than zero-shot prediction | CLIP · IllusionBench-IN unseen test domains | accuracy: 22.36\% | Helped |
| use DPLCLIP prompt learning for domain generalization | CLIP · IllusionBench-IN unseen test domains | accuracy: 13.62\% | Hurt |
| include a demonstration of the test scene type in the context (ICL3 instead of ICL1) (ablation) | Qwen-VL-Chat · IllusionBench-IN, scene prediction, 4-shot | scene accuracy: 51.4\% -> 88.1\% | Helped |
| include the test shape type in the context (ICL2) (ablation) | LLaVA · IllusionBench, shape prediction, 1-shot | shape accuracy: 97.9\% | Helped |
| include both test shape and scene types separately in the context (ICL4) (ablation) | LLaVA · IllusionBench, shape prediction, 1-shot | shape accuracy: 99.9\% | Helped |
| increase the number of in-context demonstrations (ablation) | VLMs evaluated on IllusionBench · IllusionBench | shape and scene recall | Mixed |
| use in-context learning rather than zero-shot prompting | VLMs evaluated on IllusionBench · IllusionBench | shape recall | No clear effect |
Surprising findings
- Mixture-of-experts VLMs show no advantage in shape or scene recall over individual models or closed-source models.
- In-context learning does not substantially improve abstract shape recognition or reduce scene-prediction bias across 0-4 shots.
Limitations
- Only a single shape is hidden in each image.
- Zero-shot evaluation uses prompt engineering and closed option sets, so results may depend on prompts.
- LOGO results are normalized to samples whose raw shapes the VLM recognizes, and MMICL/LLaVA ICL results show copying effects.
Experimental setup
Datasets: Stylized-ImageNet IllusionBench-IN IllusionBench-LOGO IllusionBench-ICON
Benchmarks: IllusionBench-IN IllusionBench-LOGO IllusionBench-ICON Stylized-ImageNet
Metrics: shape recall scene recall accuracy
Baselines: LLaVA-Next Qwen-VL-Chat Otter-MPT IDEFICS-9B-Instruct MMICL-T5-XXL ERM MixUp RegMixUp GroupDRO VREx DPLCLIP
Builds on: ControlNet Stylized-ImageNet CLIP MixUp DPLCLIP
Closest papers by meaning
- Visual Anagrams Reveal Hidden Differences in Holistic Shape Processing Across Vision Models (NeurIPS 2025 · similarity 0.89)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.88)
- GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Models (ICML 2026 · similarity 0.88)
- Can We Talk Models Into Seeing the World Differently? (ICLR 2025 · similarity 0.87)
- Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.87)
- Evaluating Multiview Object Consistency in Humans and Image Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.