AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs

Sirnam Swetha, Jinyu Yang, Tal Neiman, Mamshad Nayeem Rizve, Son Tran, Benjamin Yao, Trishul A Chilimbi, Mubarak Shah

ECCV 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 15 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
naively concatenate CLIP and MAE features as Q-Former input (ablation)BLIP-2 · VQAv2/GQA/OK-VQAaccuracy: VQAv2 52.4 -> 52.3; GQA 33.1 -> 32.1No clear effect
add early cross-attention to MAE inside Q-Former (ablation)BLIP-2 · VQAv2/GQA/OK-VQAaccuracy: VQAv2 52.4 -> 53.8; GQA 33.1 -> 32.7Mixed
replace early cross-attention with X-Former dual cross-attention and late MAE interaction (ablation)Q-Former with CLIP and MAE · GQAaccuracy: 32.7 -> 34.9Improved
use MAE features instead of early CLIP layer features (ablation)X-Former · VQAv2accuracy: 53.7 -> 55.0Improved
add reconstruction loss in pre-training (ablation)X-Former · VQAv2accuracy: 33.1 -> 55.0Improved
add reconstruction loss in Stage 2 LLM alignment (ablation)X-Former · VQAv2accuracy: 55.0 -> 52.4Hurt
train X-Former in Stage 2 instead of keeping it frozen (ablation)X-Former · GQAaccuracy: 25.5 -> 31.9Improved
replace X-Former with BLIP-2 style Q-Former on COCO captioningX-Former · COCO captioning zero-shotCIDEr: 131.0 -> 131.1No clear effect
add MAE branch via X-Former for object countingBLIP-2 · Object Counting COCOaccuracy: 25.88 -> 39.64Improved
scale pretraining data to 105MX-Former · GQA OPT-2.7Baccuracy: 34.1 -> 34.3Improved
add X-Former compared with BLIP-2 on Flickr30k image retrievalBLIP-2 · Flickr30k IR@1IR@1: 83.5 -> 83.3No clear effect

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy