Research map / Vision-language and multimodal / Visual question answering
Ranked from Within: Ranking Large Multimodal Models Without Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether pre-trained large multimodal models can be ranked on unlabeled target data using uncertainty scores (token NLL, entropy, sample consistency, ATC) instead of labels, evaluated across 45-47 LMMs and 8-9 VQA benchmarks.
Problem
Ranking pre-trained LMMs for a new target dataset without ground-truth labels; existing benchmark-based rankings are unstable across domains.
Approach
Use the model's own output uncertainty (softmax NLL/entropy over generated tokens, sample-based self-consistency, or ATC thresholds from a proxy labeled set) as a label-free proxy for true performance, finding sequence-level NLL to be the most robust.
Key results
- NLL_max achieves Spearman rho of 0.76 averaged across benchmarks, versus 0.60 for AoL-style benchmark transfer (AoL average rho 0.60 in Table 1 table averages).
- NLL_max reaches rho 0.94 on ChartQA and 0.91 on AI2D, while AoL reaches 0.30 on OCRVQA.
- Labeling 50 target samples gives rho 0.35 on MMMU versus 0.56 for NLL_max.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NLLmax instead of AoL for label-free model ranking | LMM ranking across benchmarks · Average across eight benchmarks | Spearman's rank correlation (ρ): 0.60 -> 0.76 | Helped |
| Use negative log-likelihood rather than normalized entropy | LMM ranking across benchmarks · Seven of eight benchmarks | ranking correlation | Helped |
| Use the first-token uncertainty score for multiple-choice questions | LMM ranking · MCVQ | ranking correlation | Helped |
| Use the penultimate-token uncertainty score for open-ended answers | LMM ranking · VQA | ranking correlation | Helped |
| Aggregate uncertainty across all generated tokens rather than using a single token | LMM ranking · Across MCVQ and VQA | ranking stability | Helped |
| Use Sample*BERT with the full answer text rather than SampleBERT on single-letter answers | LMM ranking · Four MCVQ benchmarks | Spearman's rank correlation (ρ): 0.66 -> 0.73 | Helped |
| Use stochastic sample-based scores | LMM ranking · VQA compared with MCVQ | ranking correlation | Mixed |
| Use AoL as the sole proxy for a new target domain | LMM ranking · Seven of eight benchmarks | ranking correlation | Hurt |
| Use a different proxy dataset to set the ATC threshold | ATC model ranking · Target domains | correlation with model performance | Mixed |
| Use 50 labeled target-domain samples instead of NLLmax | LLaVA-prismatic ranking · MMMU | Spearman's rank correlation (ρ): 0.56 -> 0.35 | Hurt |
Surprising findings
- Benchmark-to-benchmark rankings of LMMs correlate weakly and inconsistently, so accuracy-on-the-line transfer is unreliable for LMMs.
- Considering all generated tokens can inflate confidence because models append full answer text after the option letter, lowering uncertainty for verbose responses.
Limitations
- Correlations vary by task type; MCVQ rankings are harder, and sample-based methods are weaker on multiple-choice tasks.
- ATC depends strongly on the choice of proxy dataset due to scale differences in uncertainty.
- Entropy normalization is sensitive to vocabulary size differences across models.
Experimental setup
Datasets: ScienceQA AI2D ChartQA OCRVQA TextVQA DocVQA RealWorldQA MMMU GQA
Benchmarks: ScienceQA-IMG AI2D RealWorldQA MMMU GQA ChartQA OCRVQA TextVQA DocVQA
Metrics: Spearman rho weighted Kendall tau_w
Baselines: AoL ATC Sample-BLEU Sample-BERT labeled 50-sample subset
Builds on: accuracy-on-the-line average thresholded confidence semantic entropy BERTScore
Closest papers by meaning
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (ICML 2024 · similarity 0.87)
- On the Robustness of Large Multimodal Models Against Image Adversarial Attacks (CVPR 2024 · similarity 0.87)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.87)
- MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.87)
- F-LMM: Grounding Frozen Large Multimodal Models (CVPR 2025 · similarity 0.87)
- UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.