Research map / Vision-language and multimodal / Cross-modal learning

QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self-Penalization

NeurIPS 2024 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes QUEST, a quadruple contrastive framework that splits encoder outputs into shared and unique decoders, aligns them via quaternion cross-product spaces with orthogonal constraints, and applies shared-information-guided self-penalization to reweight hard negatives.

Problem

Standard multimodal contrastive learning treats all negatives equally and, in multi-view settings, prioritizes shared information while suppressing modality-specific unique information (feature suppression and shortcut learning).

Approach

Separate shared and unique decoders on a shared encoder; build a quaternion-style embedding via cross products of shared and unique vectors, contrast these normal vectors across modalities, add an orthogonalized cosine loss, and penalize hard negatives using a similarity-based weight matrix derived from shared representations.

Key results

• On Flickr30k, QUEST with CLIP improves RSUM by 3.2 over InfoNCE (i2t R@1 +2.4, t2i R@1 +1.5), and on MS-COCO RSUM improves by 8.1.
• On synthetic shortcut data, the paper reports an average 97.95% outperformance over baselines for CLIP.
• On image-audio and text-audio retrieval (FMA, GTZAN, Clotho, AudioCaps), QUEST beats InfoNCE on all reported RSUM values, e.g., Clotho t2a RSUM 142.41 vs 138.02.

Measured effects reported in the paper

Surprising findings

• Using a unique decoder replaced by a second shared decoder degrades performance, which the authors term unique decoder degeneration.
• Standard InfoNCE on synthetic shortcut CLIP data collapses to near-zero recall (RSUM 0.6 on MS-COCO shortcut setting).

Limitations

• Cross-product construction is limited in high-dimensional spaces, complicating control and interpretability.
• Experiments are supervised; the authors note extension to self-supervised settings is future work.
• Audio experiments use only linear decoders without modality fusion or pretraining.

Experimental setup

Datasets: Flickr30k MS-COCO FMA GTZAN CLOTHO AUDIOCAPS synthetic shortcut MNIST-augmented Flickr30k/MS-COCO

Benchmarks: Flickr30k MS-COCO FMA GTZAN CLOTHO AUDIOCAPS

Metrics: Recall@1 Recall@5 Recall@10 RSUM

Baselines: InfoNCE LTD IFM VSE++

Builds on: InfoNCE FactorCL latent target decoding implicit feature modification SimCLR

Closest papers by meaning

• Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.88)
• Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.88)
• What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.88)
• ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.86)
• On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.86)
• Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.86)