Research map / Vision-language and multimodal / Multimodal LLMs
Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dysca, a dynamic, scalable benchmark that uses Stable Diffusion XL and rule-based prompt/QA generation to evaluate LVLM perception over 20 subtasks, 51 styles, 4 scenarios and 3 question types, avoiding data leakage.
Problem
Existing LVLM perception benchmarks select images from existing datasets, risking data leakage, and largely ignore stylized images and noisy scenarios (corruption, print attacks, adversarial attacks).
Approach
Decouple image prompts into style, attribute, foreground and background metadata, generate images with text-to-image diffusion, derive QA pairs by rules (since the prompt encodes the answer), clean with CLIP and PP-OCRv3, and add controlled perturbations for noisy scenarios.
Key results
- GLM-4V achieves the top clean score of 82.09 among evaluated LVLMs, while all models remain below 85.
- Print attacking causes large drops, e.g., GLM-4V falls 36.5% (82.09 to 52.09) and GPT-4o falls 25.8% (75.69 to 56.10); corruption changes scores by under 1%.
- Rank correlation with MMBench and OCRBench is high (rho 0.70 and 0.90 on all styles); on realistic styles SeedBench-2 correlation rises from 0.46 to 0.64.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase MiniGPT-4's language model from Vicuna-7B to Vicuna-13B | MiniGPT-4 · Dysca clean scenario | score: +8.79 | Helped |
| Evaluate in the corruption scenario instead of clean | LVLMs · Dysca corruption scenario | score: less than 1% | No clear effect |
| Evaluate in print attacking instead of clean | GLM-4V · Dysca print attacking scenario | score: 82.09 to 52.09 | Hurt |
| Evaluate in adversarial attacking instead of clean | XComposer-VL · Dysca adversarial attacking scenario | score: 71.40 to 30.28 | Hurt |
| Evaluate on realistic styles only rather than all styles | Dysca · SeedBench-2 ranking correlation | Spearman's rank correlation coefficient: 0.46 vs. 0.64 | Helped |
| Remove incorrectly generated images from the evaluation set | Dysca evaluation · Dysca clean scenario | score: 82.09 -> 82.13 | No clear effect |
Surprising findings
- Blind (text-only) evaluation yields near-random scores for all LVLMs, indicating limited language-prior shortcuts.
- Models with the shared EVA-CLIP encoder (BLIP2, InstructBLIP, XComposer-VL) suffer large drops under black-box adversarial attacks, some below random chance.
Limitations
- Rule-based generation cannot easily produce cognition-level questions such as commonsense reasoning.
- Residual generation errors: manual check found 2.2% incorrect images in a 10% sample.
- Adversarial attacks use a white-box proxy (InstructBLIP) transferred to black-box models.
Experimental setup
Datasets: ImageNet IMDb rating lists community style lists
Benchmarks: Dysca
Metrics: accuracy sentence-transformer similarity sensitivity to question type (SQ) sensitivity to covariate shift (SC) Spearman rho Kendall tau KMMD
Baselines: MMBench OCRBench SEED-Bench-2 MME MMVet JourneyDB
Builds on: Stable Diffusion XL PGD JourneyDB MME SentenceTransformer
Closest papers by meaning
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
- MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs (ICML 2026 · similarity 0.90)
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.89)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
- Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping (ICLR 2025 · similarity 0.89)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.