Research map / Vision-language and multimodal / Multimodal LLMs
Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Machine Vision Therapy, which uses MLLMs with a transition-matrix-guided denoising in-context learning strategy to rectify CLIP/EVA predictions on OOD data and fine-tunes vision models on the corrected labels without human annotation.
Problem
Pre-trained vision models such as CLIP and EVA lose accuracy under out-of-distribution shifts, and fixing this usually needs human labels; MLLMs can recognize images but struggle to output exact class names for classification.
Approach
Estimate a transition matrix from the vision model's confidence-ranked samples plus a small labeled support set, then prompt an MLLM with a positive exemplar and a confusing negative exemplar (denoising ICL) to diagnose and correct predictions, and fine-tune the vision model on the MLLM's corrected labels.
Key results
- With ViT-L and MMICL, MVT+FT reaches 68.8% on ImageNet-V vs 51.8% for CLIP ViT-L, and 75.1% on ImageNet-A vs 69.3%.
- On iWildCam, MVT with MMICL achieves 25.0% vs 13.4% for CLIP ViT-L (fine-tuning not applied there).
- Transition-matrix-based selection of noisy classes beats top-N prediction selection on all six ImageNet-family datasets (e.g., IN-A 65.5 vs 60.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CLIP ViT-L using MVT-corrected predictions | CLIP ViT-L · ImageNet-V | accuracy: 53.6 -> 68.8 | Helped |
| Use MVT without fine-tuning | CLIP ViT-L · iWildCam | accuracy: 15.2 -> 25.0 | Helped |
| Replace confidence-based sampling with random sampling (ablation) | Transition-matrix estimation · ImageNet-V | L2 norm error: 3.83 -> 4.46 | Hurt |
| Use transition-matrix noisy classes instead of top-N predictions as therapy candidates (ablation) | MVT · ImageNet-A, ImageNet-Sketch, ImageNet-V | accuracy: 60.3 -> 65.5; 58.4 -> 59.0; 58.3 -> 61.6 | Helped |
| Vary the number of selected noisy classes from 6 (ablation) | MVT · ImageNet-R, ImageNet-V, ImageNet-Sketch | accuracy | Mixed |
| Increase the number of retrieved exemplar pairs (ablation) | DICL · ImageNet-R, ImageNet-V, ImageNet-Sketch | accuracy | Helped |
| Use least-similar instead of most-similar retrieved exemplars (ablation) | DICL · OOD datasets | accuracy | Hurt |
| Use logit-based instead of feature-based exemplar retrieval (ablation) | DICL · OOD datasets | accuracy | Helped |
| Retrieve exemplars from a support set with distribution shift (ablation) | DICL · DomainBed domain-shift evaluation | accuracy | No clear effect |
| Increase in-context exemplar-pair length beyond 4 (ablation) | DICL · OOD datasets | accuracy | Hurt |
| Combine vision-model confidence and MLLM diagnosing confidence into detection score Δ (ablation) | MVT · Open-class OOD detection | F1 score | Helped |
| Use Otter instead of MMICL as the MLLM backbone (ablation) | MVT · ImageNet-A and ImageNet-V | accuracy: 71.2 vs 64.1; 62.1 vs 51.9 | Mixed |
Surprising findings
- Vanilla VQA prompting of MLLMs performs far below CLIP on ImageNet-family datasets (e.g., 64.9 vs 75.8 on IN-Val for ViT-L), showing MLLMs struggle with direct classification.
- Logit-based exemplar retrieval outperforms feature-similarity retrieval, which the authors attribute to classification depending more on logits than feature similarity.
Limitations
- Relies on a labeled support set (3 images per class) and OOD test data, so some manual annotation is still needed despite the 'nearly unsupervised' claim.
- MLLM inference is slow and memory-heavy, and fine-tuning is skipped on iWildCam because most MLLM predictions there are wrong.
- Performance varies notably across MLLM backbones (MMICL vs Otter) and degrades for in-context lengths beyond 4.
Experimental setup
Datasets: ImageNet-1K ImageNet-V2 CIFAR10 CIFAR100 MNIST ImageNet-A ImageNet-R ImageNet-Sketch ImageNet-V iWildCam VLCS PACS
Benchmarks: ImageNet-Val ImageNet-V2 CIFAR10 CIFAR100 MNIST ImageNet-A ImageNet-R ImageNet-Sketch ImageNet-V iWildCam VLCS PACS
Metrics: classification accuracy F1 score (OOD detection)
Baselines: CLIP EVA VQA Vanilla FT Top-N prediction selection
Builds on: CLIP EVA RICES learning with noisy labels in-context learning theory of Xie et al.
Closest papers by meaning
- Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.90)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.90)
- Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.