Research map / Large language models / Alignment and preferences
Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a preference-based RL (ranked DPO) method that aligns a CLIP-style retrieval model with human aesthetics by distilling LLM query rephrasing and aesthetic re-ranker signals, plus two aesthetic retrieval benchmarks (HPIR and a GPT-4V win-rate judge).
Problem
Vision-language retrieval models trained on noisy web data often return images that match the query semantically but are visually unappealing or misaligned with user aesthetic intent, and existing retrieval benchmarks do not evaluate aesthetics.
Approach
Use LLM query rephrasing to enrich aesthetic expectations, build partially ordered retrieval sequences ranked by aesthetic and semantic re-rankers, and fine-tune the retrieval model with a ranked DPO loss plus pretraining loss so it internalizes these aesthetic preferences in a single-stage system.
Key results
- Ours-RLFT reaches 71.7% HPIR accuracy and 67.6% HPIR aesthetic versus 66.2%/59.4% for its pretrained version, while retrieval metrics stay near baseline (ImageNet1K-ZS 82.2, MSCOCO 40.2).
- DataComp + RLFT improves HPIR aesthetic from 62.7 to 71.9 with MSCOCO 45.1 and Flickr30K 72.2.
- In system-level GPT-4V comparisons, Ours-FT with LLM rephrasing achieves 62.7% win-and-similar rate on accuracy and aesthetic against Getty search, and 45.3%/57.3% against Bing image search.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ranked DPO preference alignment fine-tuning | Ours-PT · HPIR Aesthetic | aesthetic (%): 59.4 -> 67.6 | Helped |
| Apply ranked DPO preference alignment fine-tuning | Ours-PT · MSCOCO | Recall@1 (%): 40.2 -> 40.2 | No clear effect |
| Apply ranked DPO preference alignment fine-tuning | Ours-PT · Flickr30K | Recall@1 (%): 66.5 -> 66.4 | No clear effect |
| Apply ranked DPO preference alignment fine-tuning | CLIP · HPIR Aesthetic | aesthetic (%): 62.1 -> 71.7 | Helped |
| Apply ranked DPO preference alignment fine-tuning | DataComp · HPIR Aesthetic | aesthetic (%): 62.7 -> 71.9 | Helped |
| Use LLM query rephrasing with <k list> rather than the original query | retrieval system · HPIR Aesthetic | aesthetic (%): 59.36 -> 72.42 | Helped |
| Use LLM query rephrasing with <detail> rather than the original query | retrieval system · GPT-4V win rate, aesthetic | win rate (%): 68.75 | Helped |
| Repeat the original query to extend its length | retrieval system · HPIR Aesthetic | aesthetic (%): 59.36 -> 65.67 | Helped |
| Replace ranked DPO with RRHF loss (ablation) | alignment fine-tuning · HPIR Aesthetic | aesthetic (%): 59.4 -> 62.2 | Helped |
| Replace ranked DPO with ranked IPO loss (ablation) | alignment fine-tuning · HPIR Aesthetic | aesthetic (%): 59.4 -> 66.8 | Helped |
| Replace ranked DPO with ranked IPO loss (ablation) | alignment fine-tuning · HPIR Aesthetic | aesthetic (%): 67.6 -> 66.8 | Hurt |
| Add auto augmentation (ablation) | alignment fine-tuning · HPIR Accuracy | accuracy (%): 71.7 -> 74.4 | Helped |
Surprising findings
- Simply repeating the query text N times raises aesthetic scores of retrieved images, suggesting query length alone contributes to aesthetic gains.
- Strong data augmentation improves HPIR accuracy but harms aesthetic performance.
Limitations
- Evaluation relies on a 150-query HPIR set and a GPT-4V judge, and human labeling is subjective, so the evaluation is narrow.
- Pretraining uses a private dataset (SCVL) and a large budget, so the comparisons with CLIP and DataComp are not fully controlled; the model is weaker than DataComp on MSCOCO and Flickr30K.
- Relies on GPT-3.5-turbo for query generation and rephrasing and on existing aesthetic models as reward signals, so biases of those models propagate.
Experimental setup
Datasets: DataComp ImageNet-22k SCVL (self-collected image-text pairs) ImageNet1K MSCOCO Flickr30k
Benchmarks: ImageNet1K zero-shot MSCOCO T2I R@1 Flickr30K T2I R@1 HPIR
Metrics: accuracy (confidence-weighted) aesthetic (confidence-weighted) win rate win-and-similar rate
Baselines: CLIP DataComp Bing image search Getty search Ours-PT + Rerank
Builds on: DPO CLIP RRHF IPO InstructGPT
Closest papers by meaning
- Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization (ICLR 2026 · similarity 0.89)
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.88)
- Bridging Cognitive Gap: Hierarchical Description Learning for Artistic Image Aesthetics Assessment (AAAI 2026 · similarity 0.88)
- Learning Multi-Dimensional Human Preference for Text-to-Image Generation (CVPR 2024 · similarity 0.88)
- The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.87)
- Aligning Visual Contrastive learning models via Preference Optimization (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.