Research map / Vision-language and multimodal / CLIP and zero-shot
VILA: Learning Image Aesthetics From User Comments With Vision-Language Pretraining
Junjie Ke, Keren Ye, Jiahui Yu, Yonghui Wu, Peyman Milanfar, Feng Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on aesthetic image-comment pairs in addition to general image-text data (ablation) | VILA-P · AVA | zero-shot SRCC: 0.228 -> 0.657 | Improved |
| Add both general and aesthetic pretraining before the adapter (ablation) | VILA-R · AVA | SRCC: 0.566 -> 0.774 | Improved |
| Use a rank-based target instead of L2 score regression (ablation) | VILA-R · AVA | SRCC: 0.757 -> 0.774 | Improved |
| Use a rank-based target instead of EMD loss (ablation) | VILA-R · AVA | SRCC: 0.759 -> 0.774 | Improved |
| Use the frozen text prompt embedding as the ranking anchor (ablation) | VILA-R · AVA | SRCC: 0.763 -> 0.774 | Improved |
| Add a residual connection to the learnable image projection (ablation) | VILA-R · AVA | SRCC: 0.766 -> 0.774 | Improved |
| Finetune the image encoder rather than keep it frozen (ablation) | VILA-R · AVA | SRCC: 0.774 -> 0.780 | Improved |
| Finetune the image encoder (ablation) | VILA-P · AVA-Style | mAP: 69.0% -> 26.3% | Hurt |
| Set the contrastive-to-generative loss ratio to 1:2 (ablation) | VILA-P · AVA | zero-shot SRCC: 0.645 -> 0.657 | Improved |
| Set the ranking margin to 0.1 (ablation) | VILA-R · AVA | SRCC: 0.769 -> 0.774 | Improved |
| Randomly sample a comment for each image during aesthetic pretraining (ablation) | VILA-P · AVA | zero-shot PLCC: 0.596 -> 0.663 | Improved |
| Use ensemble prompts instead of a single prompt (ablation) | VILA-P · AVA | zero-shot SRCC: 0.605 -> 0.657 | Improved |
Later papers that compared with this method
14 later accepted papers report VILA1.5-13B as a baseline in their results tables; 12 of them report a better number on most of the results they share with it.
- ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Models (CVPR 2026 · better on 1 of 1 shared result)
- ViCToR: Improving Visual Comprehension via Token Reconstruction for Pretraining LMMs (AAAI 2026 · better on 1 of 1 shared result)
- Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving (IJCV 2026 · better on 1 of 1 shared result)
- HAMSTER: Hierarchical Action Models for Open-World Robot Manipulation (ICLR 2025 · better on 6 of 11 shared results)
- WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models (ICML 2025 · better on 5 of 5 shared results)
- UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence (ICCV 2025 · better on 3 of 5 shared results)
- Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning (AAAI 2025 · better on 4 of 4 shared results)
- Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding (CVPR 2025 · better on 2 of 2 shared results)
6 more not listed.
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning (AAAI 2025 · similarity 0.92)
- Bridging Cognitive Gap: Hierarchical Description Learning for Artistic Image Aesthetics Assessment (AAAI 2026 · similarity 0.90)
- ArtiMuse: Fine-Grained Image Aesthetics Assessment with Joint Scoring and Expert-Level Understanding (CVPR 2026 · similarity 0.89)
- Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms (NeurIPS 2024 · similarity 0.88)
- Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization (ICLR 2026 · similarity 0.88)
- UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texture (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
