Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Can We Predict Performance of Large Models across Vision-Language Tasks?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames LVLM evaluation cost reduction as matrix completion: probabilistic matrix factorization with MCMC predicts unknown model-task scores, with uncertainty used to prioritize evaluations and several enhancements for sparse data.
Problem
Evaluating many large vision-language models across many benchmarks is computationally expensive; the goal is to predict unobserved model-dataset performance scores from sparse observed scores.
Approach
Applies probabilistic matrix factorization (PMF) with MCMC to a sparse model-by-dataset score matrix, using MCMC uncertainty to rank which model-dataset pairs to evaluate next, and adds tensor factorization for multiple metrics, Bayesian priors (LKJ), and model/dataset profiles to handle sparsity.
Key results
- PMF significantly outperforms Global Mean and Mean of Means baselines when test ratio is below 90%; overall RMSE 0.193 at 20% test ratio vs 0.367 for Global Mean (Table 1 / appendix).
- Uncertainty-guided active evaluation consistently beats random selection for a fixed evaluation budget, especially when extra data is under 30%.
- With 10% core samples, combining coreset methods with PMF uncertainty (Unc) lowers overall RMSE from 0.224 (random) to 0.139.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PMF instead of mean-based prediction | LVLM performance matrix · Performance matrix, test ratio 20% | RMSE: 0.319 -> 0.192 | Helped |
| Use PMF instead of DMF | LVLM performance prediction · Performance matrix, test ratio 20%, overall | RMSE: 0.225 -> 0.193 | Helped |
| Model accuracy and BART scores jointly with PTF rather than separately with PMF (ablation) | PMF · Performance matrix, test ratio 20%, overall | RMSE: 0.175 -> 0.205 | Hurt |
| Model accuracy and BART scores jointly with PTF rather than separately with PMF (ablation) | PMF · Performance matrix, test ratio 90%, overall | RMSE: 0.327 -> 0.290 | Helped |
| Combine accuracy and BART scores into one matrix (ablation) | PMF · Performance matrix, test ratio 20%, overall | RMSE: 0.175 -> 0.193 | Hurt |
| Add Bayesian and constrained modeling (ablation) | PMF · New models, 20% performance observed | RMSE: 0.326 -> 0.297 | Helped |
| Add Bayesian and constrained modeling (ablation) | PMF · New datasets, 20% performance observed | RMSE: 0.032 -> 0.033 | Hurt |
| Add model and dataset profiles (ablation) | PTF · Sparse performance matrix | RMSE | Helped |
| Use custom model profiles (ablation) | PTF with model and dataset profiles · Sparse performance matrix | RMSE | Helped |
| Use dataset profiles (ablation) | PTF with model and dataset profiles · Sparse performance matrix | RMSE | No clear effect |
| Prioritize high-uncertainty model-dataset pairs for evaluation instead of random pairs | Active evaluation · Performance matrix, fixed evaluation budget | RMSE improvement | Helped |
| Increase latent dimension beyond around 10 (ablation) | PMF · Performance matrix test set | RMSE | No clear effect |
Surprising findings
- PMF performs worse than predicting the mean when more than 90% of entries are masked.
- Deep matrix factorization (DMF) and GCMC underperform PMF on this matrix, attributed to limited data size.
Limitations
- Predicting performance for entirely new models or datasets with no observed scores performs worse than the Global Mean baseline.
- PMF degrades to worse-than-mean predictions when observed data is very sparse (over 90% masked).
- The PTF multi-metric model assumes a linear relationship between metrics, which fails for BART and BERT scores.
Experimental setup
Datasets: SEED-Bench-2 MME MMBench MMMU CMMMU ScienceQA CV-Bench POPE PathVQA VQA GQA OKVQA
Benchmarks: SEED-Bench-2 MME MMBench MMMU ScienceQA POPE MathVision EMMA Video-MME LongVideoBench
Metrics: RMSE MAE R^2 Spearman correlation Kendall correlation Precision@K Recall@K
Baselines: Global Mean Mean of Means Deep Matrix Factorization GCMC TinyBenchmarks Random Selection Herding K-Center Greedy
Builds on: probabilistic matrix factorization Bayesian PMF Constrained PMF HEMM
Closest papers by meaning
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.87)
- A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models (ICLR 2026 · similarity 0.87)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.87)
- Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models (ICML 2026 · similarity 0.86)
- Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories (ICML 2026 · similarity 0.86)
- Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.