Research map / Detection and segmentation / Open-set and few-shot detection
Visual Anagrams Reveal Hidden Differences in Holistic Shape Processing Across Vision Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Object-Anagram dataset and Configural Shape Score (CSS) to measure absolute configural shape sensitivity across 86 vision models, finding self-supervised and language-aligned transformers score highest and that high-CSS models rely on mid-depth long-range attention.
Problem
Shape-vs-texture bias is a relative metric that conflates shape gains with texture suppression, so it does not measure the absolute configural shape competence of vision models or humans.
Approach
Generate image pairs that share an identical multiset of 16 diffusion-generated patches but permute them into two different objects (visual anagrams), so that only global configuration can support correct classification of both views; score models by the joint accuracy on both images (CSS).
Key results
- Self-supervised and language-aligned transformers (DINOv2, SigLIP2, EVA-CLIP) occupy the top of the CSS spectrum; EVA-CLIP G/14 reaches 77.78% CSS and SigLIP ViT-B/16 reaches 77.78% vs 23.61% for supervised ViT-B/16 with similar ImageNet accuracy.
- CSS correlates with robustness to noise (r=0.81), foreground bias (r=0.76), phase dependence (r=0.73) and critical band masking (r=0.83), outperforming shape-vs-texture bias on these.
- BagNet models score near chance (1.38-5.5% CSS), ruling out local border-hacking strategies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SigLIP ViT-B/16 rather than supervised ViT-B/16 | vision model · Object-Anagram | CSS: 23.61% -> 77.78% | Helped |
| restrict attention to only local neighborhoods within radius 1 or 2 patches (ablation) | DINOv2-B/14 · Object-Anagram | CSS | Hurt |
| restrict attention to patches outside a local radius (ablation) | DINOv2-B/14 · Object-Anagram | CSS | No clear effect |
| restrict attention to only local neighborhoods (ablation) | ViT-B/16 · Object-Anagram | CSS | Hurt |
| replace learned absolute PE with relative positional embeddings (ablation) | ViT-B/16 · Object-Anagram | CSS: 23.61% -> 38.89 | Helped |
| replace learned absolute PE with RoPE (ablation) | ViT-B/16 · Object-Anagram | CSS: 23.61% -> 51.38 | Helped |
| combine RoPE with absolute PE (ablation) | ViT-B/16 · Object-Anagram | CSS: 23.61% -> 51.38 | Helped |
| replace learned absolute PE with Mixed RoPE (ablation) | ViT-B/16 · Object-Anagram | CSS: 23.61% -> 48.61 | Helped |
| combine Mixed RoPE with absolute PE (ablation) | ViT-B/16 · Object-Anagram | CSS: 23.61% -> 52.77 | Helped |
| train with stylization (ablation) | vision models · Object-Anagram | CSS | No clear effect |
| increase adversarial-training epsilon (ablation) | ResNet-50 · Object-Anagram | CSS | No clear effect |
| increase top-k activation pruning sparsity (ablation) | AlexNet, VGG-16, and ResNet-50 · Object-Anagram | CSS | No clear effect |
Surprising findings
- Stylization, adversarial training and top-k pruning raise shape-vs-texture bias but yield little or no gain in CSS.
- Object-masking the cue-conflict stimuli substantially increases shape-bias scores for standard networks, suggesting the metric is confounded by surrounding texture statistics.
Limitations
- Stimuli are constrained by the priors of the diffusion model and use a uniform black paint texture, which may introduce out-of-distribution effects.
- Dataset is modest (72 anagram pairs in the main study) and covers only nine whole-object categories; part-based compositionality is not probed.
Experimental setup
Datasets: Object Anagram Dataset (generated) ImageNet-9 Imagenette2 Cue Conflict
Benchmarks: Object Anagram Dataset Cue Conflict ImageNet-9 Imagenette2 noise robustness Critical band masking Phase dependence
Metrics: Configural Shape Score (CSS) top-1 accuracy shape-vs-texture bias (accuracy-corrected) Pearson correlation Williams's test
Baselines: BagNet-9 BagNet-17 BagNet-33 ResNet-50 stylized ViT-B/16 CLIP ViT-B/16 SigLIP ViT-B/16
Builds on: visual anagrams shape-vs-texture bias (cue conflict) BagNet
Closest papers by meaning
- Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- ImageNet-trained CNNs are not biased towards texture: Revisiting feature reliance through controlled suppression (NeurIPS 2025 · similarity 0.88)
- Contour Integration Underlies Human-Like Vision (ICML 2025 · similarity 0.87)
- Evaluating Multiview Object Consistency in Humans and Image Models (NeurIPS 2024 · similarity 0.87)
- Revisiting Visual Corruptions in LVLMs: A Shape-Texture Perspective on Model Failures (CVPR 2026 · similarity 0.86)
- Discriminating image representations with principal distortions (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.