Research map / Vision-language and multimodal / Multimodal LLMs
Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Noise Resilient Confidence Alignment (NRCA), which aligns token-level prediction confidence between clean and Gaussian-noised visual inputs to counter open-response (captioning) overfitting during multi-task MLLM fine-tuning.
Problem
Multi-task fine-tuning of multimodal LLMs degrades performance on open-response tasks such as image captioning, which is attributed to over-reliance on language priors and overfitting.
Approach
Synthesize distorted visual inputs by mixing Gaussian noise with the image features and penalize the difference in average ground-truth token confidence between noisy and clean branches, scaled by the cross-entropy loss, with the clean branch detached; this avoids storing pre-trained weights, unlike mask or L2/grafting regularizers.
Key results
- On VILA1.5-3B with E=3, NRCA reaches 83.24 average accuracy vs 79.47 for Full FT, and 83.24 vs 79.52 for L2-Reg.
- On LLaVA-1.5-7B with E=3 and Flickr30k+ScienceQA, NRCA improves Flickr30k CIDEr to 75.49 from 63.11 for Full FT.
- Resource cost of NRCA is O(B x (|xv~|+|z~|)) instead of O(|theta|) required by mask and stiffness regularizers, per the paper's resource comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add NRCA confidence alignment during multi-task fine-tuning | VILA-1.5-3B · Flickr30k with ScienceQA, E=3 | CIDEr: 70.33 -> 76.00 | Helped |
| add NRCA confidence alignment during multi-task fine-tuning | VILA-1.5-3B · COCO-Cap with ScienceQA, E=3 | CIDEr: 109.58 -> 122.27 | Helped |
| add NRCA confidence alignment during multi-task fine-tuning | VILA-1.5-3B · Flickr30k with IconQA, E=3 | CIDEr: 68.20 -> 75.98 | Helped |
| add NRCA confidence alignment during multi-task fine-tuning | VILA-1.5-3B · COCO-Cap with IconQA, E=3 | CIDEr: 108.16 -> 120.82 | Helped |
| remove confidence-alignment term (ablation) | NRCA on VILA · Flickr30k with ScienceQA, E=3 | CIDEr: 75.10 -> 70.33 | Hurt |
| remove confidence-alignment term (ablation) | NRCA on VILA · ScienceQA with Flickr30k, E=3 | Top-1 accuracy: 90.23 -> 88.60 | Hurt |
| use confidence alignment without empirical-loss weighting (ablation) | NRCA on VILA · Flickr30k with ScienceQA, E=3 | CIDEr: 70.33 -> 70.87 | Helped |
| use confidence alignment without empirical-loss weighting (ablation) | NRCA on VILA · ScienceQA with Flickr30k, E=3 | Top-1 accuracy: 88.60 -> 90.43 | Helped |
| add empirical-loss weighting to confidence alignment (ablation) | NRCA on VILA · Flickr30k with ScienceQA, E=3 | CIDEr: 70.87 -> 75.10 | Helped |
| add empirical-loss weighting to confidence alignment (ablation) | NRCA on VILA · ScienceQA with Flickr30k, E=3 | Top-1 accuracy: 90.43 -> 90.23 | Hurt |
| set noisy ratio δ to 0.5 instead of 0 (ablation) | NRCA on VILA · Flickr30k, E=3 | CIDEr: 70.33 -> 75.10 | Helped |
| set noisy ratio δ to 0.5 instead of 0 (ablation) | NRCA on VILA · ScienceQA, E=3 | Top-1 accuracy: 88.60 -> 90.23 | Helped |
Surprising findings
- Random Mask-Tuning helps on smaller VILA-3B but its gains drop on the larger LLaVA-7B, suggesting its effect is constrained optimization rather than addressing language prior dominance.
- Grafting underperforms full fine-tuning on COCO-Cap and IconQA at E=5 with VILA, showing instability of stiff penalty regularization.
Limitations
- Additional storage for noisy visual inputs and logits, scaling with batch size.
- Evaluated only on two MLLM architectures (LLaVA-1.5-7B and VILA1.5-3B) and four datasets with 10k samples each.
- Open-response vs fixed-choice split uses a label-length heuristic rather than a principled task identifier.
Experimental setup
Datasets: Flickr30k COCO-Cap ScienceQA IconQA
Benchmarks: Flickr30k COCO-Cap ScienceQA IconQA
Metrics: CIDEr Top-1 accuracy mean accuracy across tasks
Baselines: Full FT Random Mask-Tuning Magnitude Tuning Grafting L2-Reg
Builds on: LLaVA VILA Random Mask-Tuning Grafting L2-Regularization
Closest papers by meaning
- Linking Perception, Confidence and Accuracy in MLLMs (CVPR 2026 · similarity 0.90)
- Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning (ICML 2025 · similarity 0.90)
- Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- Explore How to Inject Beneficial Noise in MLLMs (AAAI 2026 · similarity 0.89)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.