Research map / Vision-language and multimodal / Multimodal LLMs
AI-generated Image Quality Assessment in Visual Communication
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AIGI-VC, a 2,500-image database of AI-generated ad images scored on information clarity and emotional interaction with coarse preference labels and fine-grained descriptions, and benchmarks IQA metrics and LMMs on it.
Problem
Existing AIGI quality assessment benchmarks focus on general generated-content quality and overlook whether AI-generated images effectively communicate messages and evoke intended emotions in real-world visual communication (advertising).
Approach
A visual-communication-oriented AIGI quality dataset that evaluates communicability along information clarity and emotional interaction, using MAP-based Thurstone pairwise preference labels (reducing comparisons by 60%) and GPT-4o-assisted, human-verified fine-grained descriptions to benchmark prediction, interpretation, and reasoning.
Key results
- GPT-4o achieves the best preference prediction accuracy among LMMs, e.g., alpha 0.7928 on information clarity (all pairs) and 0.7236 on emotional interaction, with consistency kappa 0.8687 and 0.6424.
- Among IQA metrics, PickScore leads information clarity prediction (alpha 0.7518, rho 0.6807 on all pairs), while HPSv2 is strongest on emotional interaction (alpha 0.7036).
- Open-source LMMs show poor prediction consistency (kappa mostly below 0.3), indicating order-biased responses; GPT-4o's interpretation completeness is 1.3042 vs. at most 0.4835 for open-source LMMs on the scale used.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o instead of the other evaluated models for preference prediction | AIGI-VC · Information clarity and emotional interaction, D_all and D_sub | accuracy (α): 0.7928; 0.8826; 0.7236; 0.7993 | Helped |
| Use AIGI-specific CLIP-based metrics instead of HyperIQA or open-source LMMs | AIGI-VC preference prediction · Information clarity and emotional interaction | accuracy (α) | Helped |
| Use open-source LMMs for preference prediction | AIGI-VC · Information clarity and emotional interaction | consistency (κ) | Hurt |
| Use GPT-4o instead of other LMMs for challenge-specific preference prediction | AIGI-VC · Human-object interactions, fantastical ads, and positive/negative-emotion ads | accuracy (α): 0.7630; 0.7825; 0.7568; 0.7622 | Helped |
| Use ImageReward for information clarity on fantastical ads | AIGI-VC IQA metrics · Fantastical ads, information clarity | accuracy (α): 0.8421 | Helped |
| Use HPSv2 for emotional-interaction assessment on challenge subsets | AIGI-VC IQA metrics · Human-object interactions and positive/negative-emotion ads | accuracy (α): 0.7230; 0.7216; 0.7050 | Helped |
| Use GPT-4o instead of other LMMs for preference interpretation | AIGI-VC · Preference interpretation, overall | completeness, preciseness, and relevance: 1.3273; 1.5083; 1.9138 | Helped |
| Use GPT-4o instead of other LMMs for preference reasoning | AIGI-VC · Preference reasoning, overall | completeness, preciseness, and relevance: 0.9065; 1.6026; 1.8254 | Helped |
| Rely on LMMs for complete and precise preference explanations | AIGI-VC LMM preference interpretation and reasoning · Preference interpretation and reasoning | completeness and preciseness | Hurt |
Surprising findings
- Open-source LMMs perform near chance on consistency and accuracy, with many models failing to produce stable preferences regardless of content.
- Most LMMs show high relevance but low completeness and precision in interpretation and reasoning, meaning they cover visual cues but miss or contradict golden descriptions.
Limitations
- Only two ad types and 14 topics; limited generalization beyond advertising.
- Only five text-to-image generators used; GPT-4o used both as annotation aid and as judge, which may bias evaluation.
Experimental setup
Datasets: AIGI-VC
Benchmarks: AIGI-VC
Metrics: accuracy (alpha) correlation (rho/PLCC) consistency (kappa) completeness preciseness relevance
Baselines: HyperIQA WSCNet CLIPScore AestheticScore HPSv2 ImageReward PickScore mPLUG-Owl2 LLaVA-v1.5-13B InternLM-XC.2-vl BakLLaVA Idefics2
Builds on: Thurstone's Case V model Mikels emotion model
Closest papers by meaning
- Are High-Quality AI-Generated Images More Difficult for Models to Detect? (ICML 2025 · similarity 0.88)
- A-Bench: Are LMMs Masters at Evaluating AI-generated Images? (ICLR 2025 · similarity 0.87)
- AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM (CVPR 2025 · similarity 0.87)
- ArtiMuse: Fine-Grained Image Aesthetics Assessment with Joint Scoring and Expert-Level Understanding (CVPR 2026 · similarity 0.86)
- Image Quality Assessment: From Human to Machine Preference (CVPR 2025 · similarity 0.86)
- A3: Towards Advertising Aesthetic Assessment (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.