Research map / Vision-language and multimodal / Multimodal LLMs
Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CALM, a multimodal LLM for image aesthetic assessment that adds a multi-scale feature alignment module trained with text-guided self-supervised learning on unlabeled images, then instruct-tunes for aesthetic scoring, commenting, and personalized assessment.
Problem
Existing IAA methods focus on single aesthetic tasks and MLLM-based IAA suffers from scarce and noisy labeled data and lacks multi-scale aesthetic perception.
Approach
A Multi-scale Feature Alignment Module (three two-layer Qformers at low, middle, high levels plus a thematic-level projection) between a ViT and an LLM, pre-trained with self-supervised attribute-level textual pseudo-labels generated from image augmentations instead of score pseudo-labels.
Key results
- On AVA aesthetic scoring with the same labeled data, CALM reaches PLCC 0.829 and SRCC 0.815, improving over VILA by 0.055 PLCC and 0.041 SRCC; CALM-E reaches PLCC 0.852 and SRCC 0.841 at 336 resolution.
- On AVA-Captions aesthetic commenting, CALM-E (224) achieves BLEU-4 0.121 and CIDEr 0.160 versus VILA's 0.113 and 0.076.
- On FLICKR-AES PIAA, CALM achieves SRCC 0.632, slightly above the prior best AFF (0.628); in-context CALM-In reaches 0.612.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add low-, middle-, and high-level Qformers to thematic-level projection (ablation) | CALM on AVA · AVA test | PLCC: +0.061 | Helped |
| Add low-, middle-, and high-level Qformers to thematic-level projection (ablation) | CALM on AVA · AVA test | SRCC: +0.056 | Helped |
| Increase Qformer query count from 4 to 32 (ablation) | CALM on AVA-Captions · AVA-Captions test | BLEU-1: 0.509 -> 0.556 | Helped |
| Increase Qformer query count from 32 to 64 (ablation) | CALM on AVA-Captions · AVA-Captions test | BLEU-1: 0.556 -> 0.551 | Hurt |
| Remove text-guided self-supervised pre-training (ablation) | CALM on AVA · AVA test | PLCC: 0.047 | Hurt |
| Remove text-guided self-supervised pre-training (ablation) | CALM on AVA · AVA test | SRCC: 0.041 | Hurt |
| Use quality augmentations only rather than no augmentations (ablation) | CALM self-supervised learning on AVA · AVA test | PLCC: 0.782 -> 0.789 | Helped |
| Use color augmentations only rather than no augmentations (ablation) | CALM self-supervised learning on AVA · AVA test | PLCC: 0.782 -> 0.789 | Helped |
| Use subject augmentations only rather than no augmentations (ablation) | CALM self-supervised learning on AVA · AVA test | PLCC: 0.782 -> 0.716 | Hurt |
| Add subject augmentations to quality and color augmentations (ablation) | CALM self-supervised learning on AVA · AVA test | PLCC: 0.807 -> 0.829 | Helped |
| Add generic and constructed aesthetic QA data | CALM on aesthetic suggesting · Aesthetic suggesting (average) | accuracy: 0.50 -> 0.79 | Helped |
| Use in-context instructions with 5 images per annotator instead of standard personalization | CALM on FLICKR-AES · FLICKR-AES test | SRCC: 0.632 -> 0.612 | Hurt |
Surprising findings
- Removing the self-supervised pre-training lowers AS performance by 0.047 PLCC, showing the text-guided self-supervised stage matters more than structural changes alone.
- Subject-level augmentations alone give no benefit for AS, but combined with quality and color augmentations they improve results.
Limitations
- Fixed ViT-L/14 encoder cannot flexibly handle varying input resolutions as CNNs do, limiting resolution exploration.
- High computational burden of the approach may limit applicability.
- Aesthetic suggesting evaluation uses only 100 images with synthetic degradations and GPT-3.5 as judge.
Experimental setup
Datasets: AVA AVA-Captions FLICKR-AES AADB EVA PCCD Pexels SPAQ TAD66K LAION-CC-SBU ShareGPT4V CC
Benchmarks: AVA AVA-Captions FLICKR-AES PCCD (aesthetic suggesting, 100 images)
Metrics: PLCC SRCC BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE CIDEr METEOR accuracy
Baselines: VILA Q-Align UNIAA LLaVA-1.5 BLIP2 miniGPT4 GPT-4v CWS Yeo EAT POC AFF
Builds on: LLaVA-1.5 Q-Align VILA UNIAA Vicuna
Closest papers by meaning
- ArtiMuse: Fine-Grained Image Aesthetics Assessment with Joint Scoring and Expert-Level Understanding (CVPR 2026 · similarity 0.91)
- Bridging Cognitive Gap: Hierarchical Description Learning for Artistic Image Aesthetics Assessment (AAAI 2026 · similarity 0.90)
- Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization (ICLR 2026 · similarity 0.89)
- ELTA: An Enhancer against Long-Tail for Aesthetics-oriented Models (ICML 2024 · similarity 0.88)
- Venus: Benchmarking and Empowering Multimodal Large Language Models for Aesthetic Guidance and Cropping (CVPR 2026 · similarity 0.86)
- Can Vision–Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective. (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.