Research map / Vision-language and multimodal / Human and embodied modeling
VisAlign: Dataset for Measuring the Alignment between AI and Humans in Visual Perception
Jiyoung Lee, Seungho Kim, Seunghyun Won, Joonseok Lee, Marzyeh Ghassemi, James Thorne, Jaeseok Choi, O-Kil Kwon, Edward Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Softmax Probability with KNN abstention | Swin Transformer · VisAlign open test set | average visual alignment distance: 0.571 -> 0.523 | Improved |
| Replace Softmax Probability with Deep Ensemble | ViT · VisAlign open test set | average visual alignment distance: 0.629 -> 0.596 | Improved |
| Replace SP with Adjusted Softmax Probability | ViT · VisAlign open test set | Must-Abstain Category 4 distance: 0.793 -> 1.000 | Hurt |
| Replace SP with Adjusted Softmax Probability | ViT · VisAlign open test set | Must-Act Category 1 distance: 0.261 -> 0.208 | Improved |
| Swap model architecture (ViT, Swin, DenseNet, ConvNeXt, MLP-Mixer) | KNN/SP abstention · VisAlign open test set | average visual alignment distance: 0.522 -> 0.605 | No clear effect |
| Use distance-based abstention (MD, KNN) instead of probability-based | Swin Transformer · VisAlign open test set | Must-Abstain Category 4 distance: 0.793 -> 0.549 | Improved |
| Add ImageNet pre-training (ablation) | ViT + SP · VisAlign open test set | average visual alignment distance: 0.629 -> 0.322 | Improved |
| Add ImageNet pre-training (ablation) | Swin Transformer + SP · VisAlign open test set | reliability score RS_0: 363 -> 737 | Improved |
| Add ImageNet pre-training (ablation) | ViT + SP · VisAlign open test set | reliability score RS_0: 313 -> 710 | Improved |
| Add ImageNet pre-training (ablation) | DenseNet + SP · VisAlign open test set | average visual alignment distance: 0.562 -> 0.643 | Hurt |
| Add ImageNet pre-training (ablation) | ConvNeXt + SP · VisAlign open test set | average visual alignment distance: 0.560 -> 0.567 | No clear effect |
| Add SimCLR self-supervised pre-training (ablation) | Swin Transformer + SP · VisAlign open test set | average visual alignment distance: 0.571 -> 0.555 | Improved |
Closest papers by meaning
- Evaluating Multiview Object Consistency in Humans and Image Models (NeurIPS 2024 · similarity 0.89)
- When does perceptual alignment benefit vision representations? (NeurIPS 2024 · similarity 0.89)
- Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot (NeurIPS 2026 · similarity 0.88)
- Towards a learning theory of representation alignment (ICLR 2025 · similarity 0.87)
- Human and AI Perceptual Differences in Image Classification Errors (AAAI 2025 · similarity 0.87)
- Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering and Manipulating Human Perceptual Variability (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
