Research map / Vision-language and multimodal / Human and embodied modeling
Ranking-aware adapter for text-driven image ordering with CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reframes CLIP as a learning-to-rank model with a lightweight ranking-aware adapter (relational attention over image pairs) for text-guided image ranking across age, dating, counting, and IQA/IAA tasks.
Problem
Adapting CLIP-style vision-language models to rank multiple images by a text-specified quantitative concept (e.g., age, object count, quality), where existing methods rely on enumerated caption bins or single-image reasoning.
Approach
Replace image-text contrastive pairing with a learning-to-rank objective: a frozen CLIP encodes images and one universal ranking prompt; a lightweight cross-attention adapter with a regression head and a ranking-aware attention branch that attends over concatenated image pairs and outputs their score difference, supervised by pairwise hinge loss plus Smooth L1.
Key results
- On object count sorting (COCO-REM), the method reaches PLCC 0.624 and SRCC 0.557, versus 0.251/0.422 for fine-tuned CLIP-based CountingCLIP baseline and 0.558/0.507 for VILA-VLM.
- On historical colored image dating (HCI), MAE 0.32 vs 0.35 for NumCLIP (state of the art reported), a gain of about 9%.
- On KonIQ-10k MOS, PLCC 0.919 / SRCC 0.911, matching VILA-R (0.919/0.932 PLCC/SRCC) and outperforming CLIP-IQA+ (0.895/0.909); on Adience it is slightly behind L2RCLIP in MAE (0.36 vs 0.36 tie, accuracy 65.2 vs 66.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add pairwise ranking supervision to the regression-only LTR model (ablation) | Ranking-aware adapter · HCI | MAE: 0.402 -> 0.355 | Helped |
| Add pairwise ranking supervision to the regression-only LTR model (ablation) | Ranking-aware adapter · Object count sorting | PLCC: 0.612 -> 0.619 | Helped |
| Add ranking-aware attention to the pairwise ranking branch (ablation) | Ranking-aware adapter · HCI | MAE: 0.355 -> 0.317 | Helped |
| Add ranking-aware attention to the pairwise ranking branch (ablation) | Ranking-aware adapter · Object count sorting | SRCC: 0.536 -> 0.557 | Helped |
| Use separate split dot-product outputs instead of merging them into A · V (ablation) | Ranking-aware attention · HCI | MAE: 0.355 -> 0.317 | Helped |
| Use separate split dot-product outputs instead of merging them into A · V (ablation) | Ranking-aware attention · Object count sorting | PLCC: 0.602 -> 0.624 | Helped |
| Use explicit output subtraction instead of concatenating outputs (ablation) | Ranking-aware attention · HCI | MAE: 0.347 -> 0.317 | Helped |
| Use explicit output subtraction instead of concatenating outputs (ablation) | Ranking-aware attention · Object count sorting | PLCC: 0.609 -> 0.624 | Helped |
| Use cross-attention instead of self-attention to aggregate image-pair responses (ablation) | Ranking-aware attention · HCI | MAE: 0.339 -> 0.317 | Helped |
| Use cross-attention instead of self-attention to aggregate image-pair responses (ablation) | Ranking-aware attention · Object count sorting | SRCC: 0.545 -> 0.557 | Helped |
| Set the number of relational tokens to 16 rather than 1, 4, or 9 (ablation) | Ranking-aware attention · Object count sorting | PLCC: 0.6003, 0.6197, 0.6167 -> 0.6242 | Helped |
| Increase the number of relational tokens from 1 to 4 (ablation) | Ranking-aware attention · HCI | MAE: 0.321 -> 0.343 | Hurt |
Surprising findings
- Fine-tuned CLIP with caption-based contrastive training performs very poorly on IQA attributes (PLCC 0.245 on KonIQ-10k) and on object counting, below some zero-shot baselines.
- Simultaneous training on mixed tasks yields object count sorting PLCC 0.657, higher than single-task training (0.624), while slightly degrading other tasks.
Limitations
- Pairwise ranking-aware attention is incompatible with triplet or listwise ranking losses.
- Category-specific bias limits open-vocabulary counting; struggles with icons/paintings and with depth-of-field images.
- No ground truth for multi-attribute ranking; multi-attribute results are only qualitative.
Experimental setup
Datasets: COCO-REM Adience Historical colored images (HCI) KonIQ-10k AVA UTKFace CLEVR
Benchmarks: Adience HCI COCO-REM KonIQ-10k AVA UTKFace CLEVR
Metrics: MAE accuracy PLCC SRCC
Baselines: Zero-shot CLIP CoOp OrdinalCLIP L2RCLIP NumCLIP InstructBLIP BLIP-2 Flamingo VILA-R CLIP-IQA+ CountingCLIP MUSIQ
Builds on: CLIP CoOp OrdinalCLIP L2RCLIP CLIP-IQA
Closest papers by meaning
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.86)
- AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition (CVPR 2026 · similarity 0.85)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.85)
- Adaptive Parameter Selection for Tuning Vision-Language Models (CVPR 2025 · similarity 0.85)
- CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.