Research map / Vision-language and multimodal / Human and embodied modeling
Do Computer Vision Foundation Models Learn the Low-level Characteristics of the Human Visual System?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a psychophysics-based protocol with nine test types (contrast detection, masking, constancy) to measure how closely image encoders of 45 foundation and generative models match human low-level vision.
Problem
Whether computer vision foundation models trained on natural images learn low-level human visual system characteristics such as contrast detection, contrast masking, and contrast constancy.
Approach
Treat foundation model image encoders as black boxes and compare their feature-space responses to psychophysical stimuli (Gabors, band-limited noise) against human data, using a cosine-based angular distance (Sac) and no additional task-specific training, unlike prior CSF studies that train a contrast-discrimination head.
Key results
- DINOv2 and OpenCLIP show the strongest alignment with human contrast masking data, with the alignment stronger for phase-incoherent masking.
- Only DINOv2 and OpenCLIP roughly follow the human contrast constancy curves in supra-threshold contrast matching; other models show large instability or heavy low-frequency attenuation.
- Contrast detection alignment is poor for nearly all foundation models; ColorVideoVDP aligns better with human contrast detection and constancy than any foundation model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DINOv2 image encoders instead of the human contrast-detection model | DINOv2 · Contrast detection: spatial frequency, luminance, and stimulus area tests | alignment with human data | Mixed |
| Use foundation-model encoders on contrast detection stimuli | Foundation models · Contrast detection | alignment with human data | Hurt |
| Use DINOv2 or OpenCLIP encoders on contrast-masking stimuli | DINOv2 and OpenCLIP · Contrast masking | alignment with human data | Helped |
| Use OpenCLIP instead of other tested foundation-model encoders | OpenCLIP · Phase-incoherent contrast masking | alignment with human data | Helped |
| Use DINOv2 or OpenCLIP encoders for supra-threshold contrast matching | DINOv2 and OpenCLIP · Contrast matching | alignment with human data | Helped |
| Use DINOv2 encoders rather than other foundation models | DINOv2 · Nine low-level vision tests | overall alignment with human data | Helped |
| Use foundation-model encoders on contrast-masking stimuli rather than contrast-detection stimuli | Foundation models · Contrast masking versus contrast detection | alignment with human data | Helped |
| Use foundation-model encoders across luminance levels | Foundation models · Luminance – Gabor Achromatic | sensitivity change with luminance | Hurt |
| Use foundation-model encoders across stimulus areas | Most tested models · Area – Gabor Achromatic | spatial pooling alignment | Mixed |
| Use DINOv2 or OpenCLIP variants rather than other foundation-model variants | DINOv2 and OpenCLIP · Contrast masking and contrast matching | alignment with human data | Mixed |
Surprising findings
- Alignment scores of model variants (e.g., OpenCLIP) vary significantly and appear unrelated to model complexity or performance on high-level tasks.
- Models align better with human contrast masking than with contrast detection, which the authors attribute to masking stimuli being common in natural images.
Limitations
- Only the highest-complexity or best-performing variant of each model is shown in main results; alignment scores for variants vary widely.
- Model encoders were modified to accept floating-point inputs and the spatial-frequency-to-retina mapping (60 ppd) is arbitrary, which can shift similarity scores.
Experimental setup
Datasets: castleCSF human contrast sensitivity data Georgeson and Sullivan contrast matching data human contrast masking data
Benchmarks: contrast detection (CSF) contrast masking supra-threshold contrast matching
Metrics: Spearman rank correlation RMSE of log contrast Sac angular feature distance
Baselines: ColorVideoVDP castleCSF Georgeson and Sullivan contrast matching data
Builds on: castleCSF ColorVideoVDP
Closest papers by meaning
- Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models (CVPR 2024 · similarity 0.86)
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.85)
- Evaluating Multiview Object Consistency in Humans and Image Models (NeurIPS 2024 · similarity 0.84)
- Low-Resource Vision Challenges for Foundation Models (CVPR 2024 · similarity 0.84)
- Interpreting Object-level Foundation Models via Visual Precision Search (CVPR 2025 · similarity 0.84)
- Contour Integration Underlies Human-Like Vision (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.