Research map / Vision-language and multimodal / Cross-modal learning
Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that text-centric multimodal alignment is fragile under noise, dynamic modality order, and missing modalities, and proposes an LLM-driven adversarial prompting (perturbation) module that improves robustness over MLLMs and robust-training baselines on PetFinder, Airbnb, and Avito.
Problem
Text-centric multimodal alignment, which converts images and tables into text for LLMs, can lose robustness to noisy inputs, reordered modalities, and missing modalities, partly due to model/caption collapse.
Approach
Use an LLM as an adversary that rewrites text-transformed inputs (noise injection, information dropout, order permutation) toward the opposite label, and train the downstream model on these adversarial prompts together with a text-centric alignment pipeline (summarization, reasoning augmentation, one-shot ICL).
Key results
- Under missing-modality evaluation on Petfinder, the text-centric adversarial method dropped 10% versus 22.5% for robust training and 26.2% for MLLMs (as stated in text).
- Under dynamic-modality evaluation on Petfinder, the adversarial method dropped 5.5% versus 11.1% for MLLMs (as stated in text), though Table 1 reports different drop values.
- Table 1 reports ACC of 0.397 for the adversarial text-centric method on Petfinder vs 0.371 for Kosmos2 under dynamic order.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adversarial prompting instead of random perturbation | Text-centric multimodal alignment · PetFinder (dynamic modality) | accuracy: .390 -> .397 | Helped |
| Use adversarial prompting instead of naive text transformation | Text-centric multimodal alignment · Airbnb (dynamic modality) | MSE: .277 -> .274 | Helped |
| Use adversarial prompting instead of naive text transformation | Text-centric multimodal alignment · Avito (dynamic modality) | RMSE: .042 -> .042 | No clear effect |
| Use adversarial prompting instead of random perturbation | Text-centric multimodal alignment · PetFinder (missing modality) | accuracy: .370 -> .378 | Helped |
| Use adversarial prompting instead of naive text transformation | Text-centric multimodal alignment · Airbnb (missing modality) | MSE: .309 -> .302 | Helped |
| Use adversarial prompting instead of naive text transformation | Text-centric multimodal alignment · Avito (missing modality) | RMSE: .047 -> .046 | Helped |
| Add the alignment module (ablation) | Text-centric adversarial alignment · PetFinder (noisy modality) | accuracy: 0.3727 -> 0.4033 | Helped |
| Add the alignment module (ablation) | Text-centric adversarial alignment · PetFinder (dynamic modality) | accuracy: 0.383 -> 0.397 | Helped |
| Add the alignment module (ablation) | Text-centric adversarial alignment · PetFinder (missing modality) | accuracy: 0.363 -> 0.378 | Helped |
| Add the perturbation module (ablation) | Text-centric adversarial alignment · PetFinder (noisy modality) | accuracy: 0.3659 -> 0.4033 | Helped |
| Add the perturbation module (ablation) | Text-centric adversarial alignment · PetFinder (dynamic modality) | accuracy: 0.386 -> 0.397 | Helped |
| Add the perturbation module (ablation) | Text-centric adversarial alignment · PetFinder (missing modality) | accuracy: 0.362 -> 0.378 | Helped |
Surprising findings
- Text-centric naive alignment is reported as less robust than traditional robust training methods, contrary to the expectation that text abstraction would help.
- Robust training techniques performed close to random guessing under dynamic modality order.
Limitations
- Relies on proprietary GPT-4-Vision and GPT-4o APIs for captioning and reasoning, limiting reproducibility.
- Reported drop-ratio numbers in text are inconsistent with the tables.
- Evaluated on only three tabular-heavy multimodal datasets.
Experimental setup
Datasets: PetFinder.my Adoption Prediction Airbnb Pricing Prediction Avito Demand Prediction
Benchmarks: PetFinder.my Adoption Prediction Airbnb Pricing Prediction Avito Demand Prediction
Metrics: Accuracy MSE RMSE drop ratio
Baselines: Kosmos-2 Flamingo Gaussian noise injection Dropout PGD adversarial training Naive text-centric Random perturbation
Builds on: TAMML MultiBench PGD adversarial training Chain-of-Thought TabLLM serialization
Closest papers by meaning
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.88)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.88)
- DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.87)
- SAE-V: Interpreting Multimodal Models for Enhanced Alignment (ICML 2025 · similarity 0.87)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.87)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.