Research map / Vision-language and multimodal / Human and embodied modeling
When does perceptual alignment benefit vision representations?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes ViT backbones (DINO, DINOv2, CLIP, OpenCLIP, SynCLR) with LoRA on NIGHTS human similarity triplets and evaluates whether the perceptually aligned representations improve general-purpose vision tasks.
Problem
Whether aligning vision representations to human perceptual similarity judgments yields general-purpose representations that transfer beyond perceptual similarity prediction, versus distorting the original representations.
Approach
Use a second pretraining stage that LoRA-finetunes pretrained ViT backbones with a triplet hinge loss on mid-level human similarity judgments (NIGHTS), and extends the global objective to patch tokens via concatenated CLS and pooled patch features for dense prediction.
Key results
- Human-aligned models outperform base backbones in 35 of 36 counting cases across 6 models.
- Human-aligned models improve over base models in 27 of 36 monocular depth cases and boost segmentation in 16 of 20 cases.
- Perceptual alignment does not improve standard VTAB natural classification; human-aligned models largely perform worse on natural tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align pretrained vision backbones to NIGHTS mid-level human similarity judgments | DINO and DINOv2 representations · Semantic segmentation benchmarks | mIoU and P.A. | Mixed |
| Align DINO to NIGHTS human similarity judgments | DINO · Monocular depth estimation | RMSE and other reported metrics: 1.034 -> 1.032; 3.817 -> 3.757; 4.900 -> 4.788 | Helped |
| Align DINOv2 to NIGHTS human similarity judgments | DINOv2 · Monocular depth estimation | RMSE: 1.003 -> 1.062; 3.460 -> 3.720; 5.200 -> 5.082 | Mixed |
| Align vision backbones to NIGHTS human similarity judgments | Vision backbones used for retrieval-augmented generation · VTAB classification with OpenFlamingo | classification accuracy | Helped |
| Align vision representations to NIGHTS human similarity judgments | Six vision backbones · FSC147, CARPK, and Clevr-Count | k-NN counting performance | Helped |
| Align vision backbones to NIGHTS human similarity judgments | Vision backbones · DeepFashion2 Consumer-to-Shop | Top-1, Top-3, and Top-5 accuracy | Helped |
| Tune on NIGHTS instead of BAPPS, THINGS, or ImageNet triplets (ablation) | Vision backbones · Counting and DeepFashion2 instance retrieval | counting RMSE and retrieval recall | Helped |
| Tune on THINGS high-level concept triplets instead of NIGHTS (ablation) | Vision backbones · Counting and instance retrieval | counting RMSE and retrieval recall | Hurt |
| Tune on BAPPS low-level distortion triplets instead of NIGHTS (ablation) | Vision backbones · Counting and instance retrieval | counting RMSE and retrieval recall | No clear effect |
| Tune on ImageNet class-boundary triplets instead of NIGHTS (ablation) | Vision backbones · Counting and instance retrieval | counting RMSE and retrieval recall | No clear effect |
| Increase NIGHTS fine-tuning beyond 1000 steps (ablation) | DeepFashion2 instance retrieval model · DeepFashion2 | retrieval performance | Hurt |
| Fine-tune on NIGHTS rather than other perceptual datasets (ablation) | DINO backbone · Semantic segmentation | segmentation performance | Helped |
Surprising findings
- Tuning on THINGS worsens performance on several downstream tasks, and alignment hurts natural-image classification on VTAB.
- Counting improves despite NIGHTS being a global image-level similarity dataset, possibly because triplets vary in object count.
Limitations
- Perceptual alignment hurts or does not help standard natural-image classification on VTAB.
- Tuning on non-mid-level datasets (THINGS, BAPPS) can degrade performance; overfitting with many finetuning steps harms downstream performance.
- Fine-grained tasks may be harmed by mid-level alignment; RAG results limited to OpenFlamingo and IDEFICS2.
Experimental setup
Datasets: NIGHTS BAPPS THINGS ImageNet FSC147 CARPK Clevr-Count DeepFashion2 Pascal VOC ADE20k Cityscapes COCO
Benchmarks: Pascal VOC ADE20k Cityscapes COCO DAVIS2017 NYUv2 4D Light Field SUN-RGBD VTAB FSC147 CARPK Clevr-Count
Metrics: mIoU pixel accuracy RMSE AbsRel classification accuracy k-NN accuracy top-k retrieval accuracy
Baselines: DINO DINOv2 CLIP OpenCLIP SynCLR DreamSim Ensemble
Builds on: DreamSim LoRA DINOv2 CLIP OpenFlamingo
Closest papers by meaning
- Evaluating Multiview Object Consistency in Humans and Image Models (NeurIPS 2024 · similarity 0.91)
- Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot (NeurIPS 2026 · similarity 0.90)
- The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.89)
- Evaluating alignment between humans and neural network representations in image-based learning tasks (NeurIPS 2024 · similarity 0.88)
- Multidimensional Observer Model and Perceptual Dimensions of Human Image Quality Assessment (NeurIPS 2026 · similarity 0.87)
- Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering and Manipulating Human Perceptual Variability (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.