Research map / Vision-language and multimodal / Human and embodied modeling
Is Your Image a Good Storyteller?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Image Semantic Assessment (ISA) task and a 2,946-image dataset with entity and semantic complexity scores, and proposes VLISA, which uses GPT-4o to extract textual features that a regression discriminator uses to predict scores.
Problem
Automatically assessing how semantically complex (story-rich) an image is, which existing image complexity, quality, and aesthetics assessments do not cover.
Approach
Use an LVLM (GPT-4o) as a feature extractor to convert images into natural-language descriptions or chain-of-reasonings, then train a discriminator (ViLT, BERT, or Longformer) to regress entity and semantic complexity scores.
Key results
- On Semantic Complexity Scoring, CoT VLISA (ViLT) reaches PCC 0.823 and SRCC 0.817, versus 0.677 PCC for ViT and 0.634 PCC for ICNet.
- On Entity Complexity Scoring, Naive VLISA (ViLT) achieves the best RMSE of 0.079 and PCC of 0.952, slightly above ViT (PCC 0.929).
- Replacing GPT-4o with the open-source CogVLM2 as feature extractor does not significantly degrade Naive VLISA (ViLT), e.g., semantic PCC 0.799.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Naive VLISA (ViLT) instead of ViT | Entity Complexity Scoring · ISA dataset test set | PCC: 0.929 (0.002) -> 0.952 (0.002) | Helped |
| Use CoT VLISA (ViLT) instead of ViT | Semantic Complexity Scoring · ISA dataset test set | PCC: 0.677 (0.003) -> 0.823 (0.002) | Helped |
| Use CoT features (CoRs and description) instead of naive description features | VLISA with ViLT · Semantic Complexity Scoring | PCC: 0.799 (0.004) -> 0.823 (0.002) | Helped |
| Use CoT features (CoRs and description) instead of naive description features | VLISA with BERT · Semantic Complexity Scoring | PCC: 0.798 (0.003) -> 0.812 (0.001) | Helped |
| Use CoT features (CoRs and description) instead of naive description features | VLISA with Longformer · Semantic Complexity Scoring | PCC: 0.800 (0.003) -> 0.804 (0.005) | Helped |
| Use Naive VLISA instead of CoT VLISA | VLISA with BERT · Entity Complexity Scoring | PCC: 0.902 (0.003) -> 0.928 (0.001) | Helped |
| Use Naive VLISA instead of CoT VLISA | VLISA with Longformer · Entity Complexity Scoring | PCC: 0.901 (0.002) -> 0.931 (0.0005) | Helped |
| Use CogVLM2 rather than GPT-4o as feature extractor | Naive VLISA (ViLT) · Entity Complexity Scoring | PCC: 0.952 (0.002) -> 0.951 (0.0005) | No clear effect |
| Use CogVLM2 rather than GPT-4o as feature extractor | Naive VLISA (ViLT) · Semantic Complexity Scoring | PCC: 0.799 (0.004) -> 0.799 (0.001) | No clear effect |
| Add CoRs to the extracted features (ablation) | CoT VLISA (ViLT) · Semantic Complexity Scoring | PCC: 0.818 (0.003) -> 0.823 (0.002) | Helped |
| Add description to the extracted features (ablation) | CoT VLISA (ViLT) · Semantic Complexity Scoring | PCC: 0.819 (0.007) -> 0.823 (0.002) | Helped |
| Add CoRs to the extracted features (ablation) | CoT VLISA (BERT) · Semantic Complexity Scoring | PCC: 0.809 (0.001) -> 0.812 (0.001) | Helped |
Surprising findings
- Naive VLISA (BERT/Longformer) outperforms CoT VLISA on the Entity Complexity Scoring task, while CoT helps on semantic scoring.
- Entity richness and semantic richness are not strongly positively correlated; images with few entities can still tell interesting stories.
Limitations
- Relies on GPT-4o features, so performance is tied to a closed-source LVLM's descriptions.
- Dataset is small (2,946 images) and drawn from Pinterest with single-source collection.
- Semantic scores are based on subjective annotation by students with criteria-driven guidelines.
Experimental setup
Datasets: ISA dataset (own, Pinterest images)
Benchmarks: ISA dataset
Metrics: RMSE RMAE PCC SRCC
Baselines: ICNet ViT
Builds on: Chain-of-Thought CogBench ICNet
Closest papers by meaning
- Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding (CVPR 2024 · similarity 0.87)
- CaptionQA: Is Your Caption as Useful as the Image Itself? (CVPR 2026 · similarity 0.86)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.85)
- What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.85)
- SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis (NeurIPS 2026 · similarity 0.85)
- Visual Data-Type Understanding does not emerge from scaling Vision-Language Models (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.