Research map / Vision-language and multimodal / Visual question answering
GLEN: Generalized Focal Loss Ensemble of Low-Rank Networks for Calibrated Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GLEN, an ensemble of low-rank factorized final layers of VQA models, each trained with a generalized focal loss to diversify members, improving calibration on VQA-v2 and AdVQA without hurting accuracy.
Problem
VQA models with large backbones are overconfident on wrong answers due to memorization, limiting reliability in critical domains; existing post-hoc calibration needs extra validation data and ensembles of similar models inherit poor calibration.
Approach
Factorize the final classification layer into a low-rank form (W≈UV) to reduce overconfidence, and ensemble several such networks each trained with a generalized focal loss (weights constrained by an f-divergence set) so members focus on different difficulty levels, giving diversity and a bias-variance trade-off interpretation.
Key results
- On LXMERT/Pythia/CLIP-ViL/ViLBERT/VisualBERT, GLEN reduces ID ECE to 0.06/0.06/0.08/0.05/0.03 vs 0.14/0.14/0.18/0.19/0.14 for baselines, with comparable accuracy.
- On VisualBERT, GLEN ECE improves about 11% over the second-best baseline; ViLBERT OOD ECE improves 10% and OC 13% over best competitor.
- On BEiT-3, GLEN reduces ECE from 0.09 to 0.02 while accuracy rises from 74.68 to 74.95.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply GLEN instead of the strongest listed calibration baseline | LXMERT · VQA-v2 test (ID) | ECE: 0.09 -> 0.06 | Helped |
| Apply GLEN instead of the strongest listed calibration baseline | Pythia · VQA-v2 test (ID) | ECE: 0.09 -> 0.06 | Helped |
| Apply GLEN instead of the strongest listed calibration baseline | CLIP-ViL · VQA-v2 test (ID) | ECE: 0.13 -> 0.08 | Helped |
| Apply GLEN instead of the strongest listed calibration baseline | ViLBERT · VQA-v2 test (ID) | ECE: 0.14 -> 0.05 | Helped |
| Apply GLEN instead of the strongest listed calibration baseline | VisualBERT · VQA-v2 test (ID) | ECE: 0.14 -> 0.03 | Helped |
| Apply GLEN rather than the baseline | BEiT-3 · VQA-v2 test | ECE: 0.09 -> 0.02 | Helped |
| Ensemble dense VQA models (ablation) | Pythia · VQA-v2 test | ECE: 0.14 -> 0.13 | Helped |
| Apply low-rank factorization to ensemble models (ablation) | Pythia · VQA-v2 test | ECE: 0.13 -> 0.09 | Helped |
| Use generalized focal loss to diversify the low-rank ensemble (ablation) | Pythia LRF Ensemble · VQA-v2 test | ECE: 0.09 -> 0.06 | Helped |
| Ensemble dense VQA models (ablation) | VisualBERT · VQA-v2 test | ECE: 0.14 -> 0.12 | Helped |
| Apply low-rank factorization to ensemble models (ablation) | VisualBERT · VQA-v2 test | ECE: 0.12 -> 0.09 | Helped |
| Use generalized focal loss to diversify the low-rank ensemble (ablation) | VisualBERT LRF Ensemble · VQA-v2 test | ECE: 0.09 -> 0.03 | Helped |
Surprising findings
- Compressing earlier layers degrades accuracy substantially, whereas compressing only the final layer improves calibration with little accuracy loss.
- A standard ensemble of low-rank final-layer models shows only about 2% ECE improvement and still overfits, because members are highly correlated.
Limitations
- Post-hoc baselines require an additional validation split, while the approach trains multiple networks (ensemble cost).
- Gains in accuracy are not uniform; GLEN sometimes slightly lowers accuracy (e.g., LXMERT ID 72.93 vs 73.06, Pythia ID 66.15 vs ensemble 66.75).
- Theoretical equivalence holds only with high probability and depends on bounded loss difference K.
Experimental setup
Datasets: VQA-v2 AdVQA
Benchmarks: VQA-v2 AdVQA
Metrics: VQA accuracy ECE over-confidence (OC) AUC C@R (C@1, C@5, C@10, C@20)
Baselines: Baseline Temperature Scaling Vector Scaling Selector Ensemble LRF Ensemble
Builds on: focal loss Selector (Whitehead et al. 2022) temperature scaling deep ensembles
Closest papers by meaning
- RMLVQA: A Margin Loss Approach for Visual Question Answering With Language Biases (CVPR 2023 · similarity 0.89)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.88)
- Tips and Tricks for Visual Question Answering: Learnings From the 2017 Challenge (CVPR 2018 · similarity 0.88)
- VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models (AAAI 2024 · similarity 0.88)
- Rethinking Token Reduction for Large Vision-Language Models (CVPR 2026 · similarity 0.87)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.