Research map / Vision-language and multimodal / Human and embodied modeling

The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Language-grounded scene text and structure · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Collects a large dataset of aspect-conditioned human odd-one-out judgments over generated image triplets and fine-tunes Qwen3-VL-8B-Embedding into TPIPS, a text-prompted perceptual similarity metric that narrows the gap to human consensus.

Problem

Existing perceptual similarity metrics output a single scalar and cannot condition on a specific visual aspect (e.g., color vs. shape), and VLMs do not match human aspect-conditioned similarity judgments.

Approach

Collect human odd-one-out votes over FLUX-generated triplets under free-form aspect conditions, then fine-tune a VLM embedding model (late, mid, or early fusion) with a softmax cross-entropy over pairwise text-conditioned similarities to produce a steerable similarity score.

Key results

• On the odd-one-out task, fine-tuning reduces the gap to human consensus (67.5%) from 9.1% to 2.8%, versus the best baseline Gemini-3.1-Pro at 58.4% agreement.
• On the external-algorithm 2AFC set, the gap to human consensus drops from 15.8% to 10.0%.
• On GeneCIS Focus Attribute R@1, TPIPS reaches 31.35 vs 30.85 for its Qwen3-VL-8B-Emb backbone; on QARE-Bench, TPIPS achieves the highest all-attribute mAP among 8B models (84.9).

Surprising findings

• Off-the-shelf LPIPS and DreamSim, which ignore the aspect, perform near or below strong VLM baselines on the aspect-conditioned task, showing overall similarity alone is insufficient.
• Early fusion gives only a modest improvement over late fusion, and late fusion remains competitive while enabling cacheable retrieval.

Limitations

• VLM-based metric is much more expensive to compute than LPIPS or DreamSim, and aspect-conditioned embedding retrieval needs per-aspect indexing.
• Concept entanglement: the model cannot fully isolate an aspect from object identity.
• Training data is synthetic and drawn from aesthetically curated FLUX-Reason-6M prompts; aspects are VLM-proposed and annotator pool is not representative.

Experimental setup

Datasets: FLUX-Reason-6M OpenImages WikiArt BAPPS NIGHTS GeneCIS QARE-Bench

Benchmarks: Odd-one-out (in-distribution test) External-algorithm 2AFC BAPPS NIGHTS GeneCIS Focus Attribute/Object QARE-Bench

Metrics: agreement with humans human consensus gap mAP R@K AIS

Baselines: LPIPS DreamSim GPT-5.4 Gemini 3.1 Pro Qwen3-VL InternVL3.5 LLaVA-OneVision VLM2Vec-2.0 Gemini-2-Embedding GeneCIS SteerViT CRL

Builds on: LPIPS DreamSim Qwen3-VL-Embedding FLUX.1-dev odd-one-out softmax choice model

Closest papers by meaning

• Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.90)
• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception (ICML 2026 · similarity 0.89)
• When does perceptual alignment benefit vision representations? (NeurIPS 2024 · similarity 0.89)
• Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding (CVPR 2024 · similarity 0.89)
• WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
• ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.88)