AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Multi-Modal Alignment Using Representation Codebook

Jiali Duan, Liqun Chen, Son Tran, Jinyu Yang, Yi Xu, Belinda Zeng, Trishul Chilimbi

CVPR 2022 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 59 citations (OpenAlex)

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add intra-modal alignment to cross-modal alignment (ablation)CODIS without codebook · MSCOCO (5K), zero-shot text retrievalR@1: 68.60 -> 69.86Improved
Add codebook loss using teacher features to cross-modal alignment only (ablation)CODIS cross-alignment-only baseline · MSCOCO (5K), zero-shot text retrievalR@1: 68.60 -> 70.74Improved
Add codebook loss using teacher features to intra- and cross-modal alignment (ablation)CODIS · MSCOCO (5K), zero-shot text retrievalR@1: 69.86 -> 71.10Improved
Use teacher rather than student features to compute codebook loss (ablation)CODIS with intra- and cross-modal alignment · MSCOCO (5K), zero-shot text retrievalR@1: 71.12 -> 71.10No clear effect
Add intra-modal alignment (ablation)CODIS zero-shot retrieval · MSCOCO (5K), zero-shot image retrievalR@1: 50.10 -> 50.52Improved
Add intra-modal alignment (ablation)CODIS zero-shot retrieval · Flickr30K (1K), zero-shot text retrievalR@1: 84.90 -> 85.80Improved
Add intra-modal alignment (ablation)CODIS zero-shot retrieval · Flickr30K (1K), zero-shot image retrievalR@1: 68.18 -> 69.70Improved
Use 500 codewords instead of the default codebook size (ablation)CODIS pretrained on MSCOCO · MSCOCO zero-shot text retrievalTR@1: 59.38 -> 55.52Hurt
Vary codebook size to 0.5x (ablation)CODIS pretrained on MSCOCO · MSCOCO zero-shot text retrievalTR@1: 59.38 -> 58.66Hurt
Vary codebook size to 2.0x (ablation)CODIS pretrained on MSCOCO · MSCOCO zero-shot text retrievalTR@1: 59.38 -> 59.02No clear effect
Vary codebook size to 3K codewords (ablation)CODIS pretrained on MSCOCO · MSCOCO zero-shot text retrievalTR@1: 59.38 -> 58.96No clear effect
Use codebook-based cluster alignment with teacher-student distillationCODIS · MSCOCO (5K), zero-shot text retrievalR@1: 68.6 -> 71.5Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy