Research map / Vision-language and multimodal / Multimodal LLMs
Robust Vision-Language Models via Manifold-Adversarial Adapters
Hao Li, Zeyu Xiao, Junhao Zhou, Peng Liu, Yang Zhao, Wei Jia
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a manifold-adversarial adapter approach for making vision-language models robust to real-world image corruptions via parameter-efficient adaptation.
Problem
Vision-language models degrade under real-world image corruptions, affecting both visual recognition and language-grounded reasoning; Euclidean feature alignment of corrupted and clean representations is argued to be insufficient.
Approach
Parameter-efficient adapters that align corrupted features with clean references using a manifold-aware (rather than purely Euclidean) alignment objective, combined with adversarial training, as an alternative to cascaded image restoration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adversarial manifold constraint to feature distillation (ablation) | MAA adapters · R-Bench Dis | score: 58.70 -> 59.39 | Improved |
| naive MSE-only adapter adaptation vs zero-shot (ablation) | LLaVA-1.6-Mistral-7B · R-Bench Dis | score: 58.79 -> 57.98 | Hurt |
| add adversarial constraint to full fine-tuning (ablation) | feature distillation · R-Bench Dis | score: 32.93 -> 34.34 | No clear effect |
| add adversarial constraint to LoRA (ablation) | feature distillation · R-Bench Dis | score: 56.77 -> 57.98 | Improved |
| add adversarial constraint to prefix tuning (ablation) | feature distillation · R-Bench Dis | score: 55.35 -> 58.99 | Improved |
| add window self-attention branch (ablation) | MAA · R-Bench Dis | score: 58.79 -> 59.39 | Improved |
| add max pooling branch (ablation) | MAA · R-Bench Dis | score: 59.39 -> 59.39 | No clear effect |
| train on hybrid multi-source data instead of single source (ablation) | MAA · R-Bench Dis | score: 58.99 -> 59.39 | Improved |
| prepend Real-ESRGAN restoration before VLM | LLaVA-1.6-Mistral-7B · MMVet | score: 42.33 -> 38.12 | Hurt |
| prepend external image restoration models before VLM | LLaVA-1.6-Mistral-7B · R-Bench Dis | score: 58.79 -> 56.97 to 58.38 | Hurt |
| use InfoNCE as feature objective (ablation) | MAA adapter · R-Bench Dis | score: 58.79 -> 0.00 | Hurt |
| apply MAA to InternVL3.5-30B-A3B | InternVL3.5-30B-A3B · R-Bench Dis | score: 71.51 -> 72.93 | Improved |
Closest papers by meaning
- An Inverse-Adversarial and Difficulty-Adaptive Robust Vision-Language Model (ECCV 2026 · similarity 0.91)
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.89)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.88)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.88)
- From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration (CVPR 2025 · similarity 0.88)
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
