AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

VisAlign: Dataset for Measuring the Alignment between AI and Humans in Visual Perception

Jiyoung Lee, Seungho Kim, Seunghyun Won, Joonseok Lee, Marzyeh Ghassemi, James Thorne, Jaeseok Choi, O-Kil Kwon, Edward Choi

NeurIPS 2023 · Vision-language and multimodal › Human and embodied modeling · approach: Human-like visual cognition in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace Softmax Probability with KNN abstentionSwin Transformer · VisAlign open test setaverage visual alignment distance: 0.571 -> 0.523Improved
Replace Softmax Probability with Deep EnsembleViT · VisAlign open test setaverage visual alignment distance: 0.629 -> 0.596Improved
Replace SP with Adjusted Softmax ProbabilityViT · VisAlign open test setMust-Abstain Category 4 distance: 0.793 -> 1.000Hurt
Replace SP with Adjusted Softmax ProbabilityViT · VisAlign open test setMust-Act Category 1 distance: 0.261 -> 0.208Improved
Swap model architecture (ViT, Swin, DenseNet, ConvNeXt, MLP-Mixer)KNN/SP abstention · VisAlign open test setaverage visual alignment distance: 0.522 -> 0.605No clear effect
Use distance-based abstention (MD, KNN) instead of probability-basedSwin Transformer · VisAlign open test setMust-Abstain Category 4 distance: 0.793 -> 0.549Improved
Add ImageNet pre-training (ablation)ViT + SP · VisAlign open test setaverage visual alignment distance: 0.629 -> 0.322Improved
Add ImageNet pre-training (ablation)Swin Transformer + SP · VisAlign open test setreliability score RS_0: 363 -> 737Improved
Add ImageNet pre-training (ablation)ViT + SP · VisAlign open test setreliability score RS_0: 313 -> 710Improved
Add ImageNet pre-training (ablation)DenseNet + SP · VisAlign open test setaverage visual alignment distance: 0.562 -> 0.643Hurt
Add ImageNet pre-training (ablation)ConvNeXt + SP · VisAlign open test setaverage visual alignment distance: 0.560 -> 0.567No clear effect
Add SimCLR self-supervised pre-training (ablation)Swin Transformer + SP · VisAlign open test setaverage visual alignment distance: 0.571 -> 0.555Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy