Research map / Vision-language and multimodal / Cross-modal learning
Learning Multimodal Data Augmentation in Feature Space
Zichang Liu, Zhiqiang Tang, Xingjian Shi, Aston Zhang, Mu Li, Anshumali Shrivastava, Andrew Gordon Wilson
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add jointly learned feature-space augmentation with adversarial training and consistency regularization | Multimodal-Net and ALBEF · Hateful Memes, Food101, Petfinder, Melbourne Airbnb, News Channel, Wine Reviews, Kick Starter Funding, and SNLI-VE | benchmark metric: 0.6939 -> 0.7562; 0.9387 -> 0.9452; 0.2911 -> 0.3537; 0.3946 -> 0.4047; 0.4754 -> 0.4798; 0.8192 -> 0.8262; 0.7571 -> 0. | Improved |
| replace joint learned augmentation with independent input augmentation | Multimodal-Net · News Channel | accuracy: 0.4754 -> 0.4745 | Hurt |
| replace joint learned augmentation with Mixup | Multimodal-Net · Wine Reviews | accuracy: 0.8192 -> 0.8143 | Hurt |
| replace joint learned augmentation with Manifold Mixup | Multimodal-Net · Melbourne Airbnb and Wine Reviews | accuracy: 0.3946 -> 0.3840; 0.8192 -> 0.8126 | Hurt |
| remove augmentation-network regularization (ablation) | LeMDA · Hateful Memes, Food101, Petfinder, Melbourne Airbnb, News Channel, Wine Reviews, and Kick Starter Funding | accuracy: 0.7433 vs 0.7562; 0.9433 vs 0.9452; 0.3369 vs 0.3537; 0.3935 vs 0.4047; 0.4851 vs 0.4869; 0.8228 vs 0.8263; 0.7609 vs 0. | Hurt |
| use consistency regularization instead of feature-space L2 regularization (ablation) | LeMDA · Hateful Memes, Food101, Petfinder, Melbourne Airbnb, News Channel, Wine Reviews, and Kick Starter Funding | accuracy: 0.7562 vs 0.7472; 0.9452 vs 0.9415; 0.3537 vs 0.3420; 0.4047 vs 0.3987; 0.4869 vs 0.4869; 0.8263 vs 0.8255; 0.7614 vs 0. | Improved |
| add feature-space L2 alongside consistency regularization (ablation) | LeMDA · Melbourne Airbnb and News Channel | accuracy: 0.4047 -> 0.4051; 0.4869 -> 0.4894 | Improved |
| replace MLP-VAE with Attention-VAE (ablation) | LeMDA · Hateful Memes, Food101, Petfinder, Melbourne Airbnb, News Channel, Wine Reviews, and Kick Starter Funding | accuracy: 0.7562 -> 0.7483; 0.9452 -> 0.9443; 0.3537 -> 0.3456; 0.4047 -> 0.4031; 0.4798 -> 0.4733; 0.8262 -> 0.8250; 0.7614 -> 0. | Hurt |
| add an augmentation network (MLP-VAE) (ablation) | Multimodal-Net · Hateful Memes, Food101, Petfinder, Melbourne Airbnb, News Channel, Wine Reviews, and Kick Starter Funding | accuracy: 0.6939 -> 0.7562; 0.9387 -> 0.9452; 0.2911 -> 0.3537; 0.3946 -> 0.4047; 0.4754 -> 0.4798; 0.8192 -> 0.8262; 0.7571 -> 0. | Improved |
| apply consistency loss to all samples by disabling confidence masking (ablation) | LeMDA · Hateful Memes | accuracy: 0.7410 vs 0.7556 | Hurt |
| set confidence-mask threshold to 0.8 instead of 0.5 (ablation) | LeMDA · Hateful Memes | accuracy: 0.7556 -> 0.7371 | Hurt |
| augment only one modality or apply independent modality-specific augmentation (ablation) | Hateful Memes multimodal model · Hateful Memes | accuracy: 0.6939 -> 0.6323 to 0.7444 | Mixed |
Closest papers by meaning
- Contrastive Visual Data Augmentation (ICML 2025 · similarity 0.87)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.87)
- Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.87)
- MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning (NeurIPS 2025 · similarity 0.86)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.86)
- UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
