Research map / Vision-language and multimodal / Multimodal LLMs
ViLU: Learning Vision-Language Uncertainties for Failure Prediction
Marc Lafon, Yannis Karmim, Julio Silva-Rodríguez, Paul Couairon, Clément Rambour, Raphael Fournier-Sniehotta, Ismail Ben Ayed, Jose Dolz, Nicolas Thome
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add predicted caption embedding to visual embedding (ablation) | ViLU · ImageNet-1k | AUC: 78.7 -> 88.8 | Improved |
| add cross-attention on top of visual embedding and predicted caption (ablation) | ViLU · CC12M | AUC: 88.9 -> 95.2 | Improved |
| add cross-attention (ablation) | ViLU · CIFAR-10 | FPR95: 10.8 -> 8.2 | Improved |
| add predicted caption to model with cross-attention (ablation) | ViLU · ImageNet-1k | AUC: 86.1 -> 89.5 | Improved |
| train as binary classification instead of MSE loss regression (ablation) | ViLU · ImageNet-1k | AUC: 84.8 -> 88.6 | Improved |
| add automatic class weighting to BCE (ablation) | ViLU · ImageNet-1k | AUC: 88.6 -> 89.5 | Improved |
| ViLU instead of MCM on image-label datasets | ViLU · 13 image-label datasets average | FPR95: 70.6 -> 29.9 | Improved |
| ViLU instead of BayesVLM on CC12M | ViLU · CC12M | FPR95: 53.3 -> 25.2 | Improved |
| LVU vision-only loss prediction applied to image-caption datasets | LVU · CC12M | FPR95: 58.8 -> 76.5 | Hurt |
| ViLU on low zero-shot accuracy dataset | ViLU · EuroSAT | AUC: 64.1 -> 98.8 | Improved |
| increase inference batch size from 128 to 32768 | ViLU · CC12M | AUC: 96.9 -> 91.1 | Hurt |
| 5-shot adaptation of ViLU on ImageNet-Sketch | ViLU · ImageNet-Sketch | FPR95: 73.1 -> 54.4 | Improved |
Later papers that compared with this method
1 later accepted paper reports ViLU as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Revisiting Confidence Calibration for Misclassification Detection in VLMs (ICLR 2026 · better on 6 of 6 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere (NeurIPS 2025 · similarity 0.90)
- Probabilistic Language-Image Pre-Training (ICLR 2025 · similarity 0.89)
- An Empirical Study Into What Matters for Calibrating Vision-Language Models (ICML 2024 · similarity 0.88)
- FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty (ICML 2026 · similarity 0.88)
- Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding (NeurIPS 2025 · similarity 0.88)
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
