AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Learning Multimodal Data Augmentation in Feature Space

Zichang Liu, Zhiqiang Tang, Xingjian Shi, Aston Zhang, Mu Li, Anshumali Shrivastava, Andrew Gordon Wilson

ICLR 2023 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 9 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add jointly learned feature-space augmentation with adversarial training and consistency regularizationMultimodal-Net and ALBEF · Hateful Memes, Food101, Petfinder, Melbourne Airbnb, News Channel, Wine Reviews, Kick Starter Funding, and SNLI-VEbenchmark metric: 0.6939 -> 0.7562; 0.9387 -> 0.9452; 0.2911 -> 0.3537; 0.3946 -> 0.4047; 0.4754 -> 0.4798; 0.8192 -> 0.8262; 0.7571 -> 0.Improved
replace joint learned augmentation with independent input augmentationMultimodal-Net · News Channelaccuracy: 0.4754 -> 0.4745Hurt
replace joint learned augmentation with MixupMultimodal-Net · Wine Reviewsaccuracy: 0.8192 -> 0.8143Hurt
replace joint learned augmentation with Manifold MixupMultimodal-Net · Melbourne Airbnb and Wine Reviewsaccuracy: 0.3946 -> 0.3840; 0.8192 -> 0.8126Hurt
remove augmentation-network regularization (ablation)LeMDA · Hateful Memes, Food101, Petfinder, Melbourne Airbnb, News Channel, Wine Reviews, and Kick Starter Fundingaccuracy: 0.7433 vs 0.7562; 0.9433 vs 0.9452; 0.3369 vs 0.3537; 0.3935 vs 0.4047; 0.4851 vs 0.4869; 0.8228 vs 0.8263; 0.7609 vs 0.Hurt
use consistency regularization instead of feature-space L2 regularization (ablation)LeMDA · Hateful Memes, Food101, Petfinder, Melbourne Airbnb, News Channel, Wine Reviews, and Kick Starter Fundingaccuracy: 0.7562 vs 0.7472; 0.9452 vs 0.9415; 0.3537 vs 0.3420; 0.4047 vs 0.3987; 0.4869 vs 0.4869; 0.8263 vs 0.8255; 0.7614 vs 0.Improved
add feature-space L2 alongside consistency regularization (ablation)LeMDA · Melbourne Airbnb and News Channelaccuracy: 0.4047 -> 0.4051; 0.4869 -> 0.4894Improved
replace MLP-VAE with Attention-VAE (ablation)LeMDA · Hateful Memes, Food101, Petfinder, Melbourne Airbnb, News Channel, Wine Reviews, and Kick Starter Fundingaccuracy: 0.7562 -> 0.7483; 0.9452 -> 0.9443; 0.3537 -> 0.3456; 0.4047 -> 0.4031; 0.4798 -> 0.4733; 0.8262 -> 0.8250; 0.7614 -> 0.Hurt
add an augmentation network (MLP-VAE) (ablation)Multimodal-Net · Hateful Memes, Food101, Petfinder, Melbourne Airbnb, News Channel, Wine Reviews, and Kick Starter Fundingaccuracy: 0.6939 -> 0.7562; 0.9387 -> 0.9452; 0.2911 -> 0.3537; 0.3946 -> 0.4047; 0.4754 -> 0.4798; 0.8192 -> 0.8262; 0.7571 -> 0.Improved
apply consistency loss to all samples by disabling confidence masking (ablation)LeMDA · Hateful Memesaccuracy: 0.7410 vs 0.7556Hurt
set confidence-mask threshold to 0.8 instead of 0.5 (ablation)LeMDA · Hateful Memesaccuracy: 0.7556 -> 0.7371Hurt
augment only one modality or apply independent modality-specific augmentation (ablation)Hateful Memes multimodal model · Hateful Memesaccuracy: 0.6939 -> 0.6323 to 0.7444Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy