Research map / Vision-language and multimodal / Multimodal LLMs
X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs
Sirnam Swetha, Jinyu Yang, Tal Neiman, Mamshad Nayeem Rizve, Son Tran, Benjamin Yao, Trishul A Chilimbi, Mubarak Shah
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| naively concatenate CLIP and MAE features as Q-Former input (ablation) | BLIP-2 · VQAv2/GQA/OK-VQA | accuracy: VQAv2 52.4 -> 52.3; GQA 33.1 -> 32.1 | No clear effect |
| add early cross-attention to MAE inside Q-Former (ablation) | BLIP-2 · VQAv2/GQA/OK-VQA | accuracy: VQAv2 52.4 -> 53.8; GQA 33.1 -> 32.7 | Mixed |
| replace early cross-attention with X-Former dual cross-attention and late MAE interaction (ablation) | Q-Former with CLIP and MAE · GQA | accuracy: 32.7 -> 34.9 | Improved |
| use MAE features instead of early CLIP layer features (ablation) | X-Former · VQAv2 | accuracy: 53.7 -> 55.0 | Improved |
| add reconstruction loss in pre-training (ablation) | X-Former · VQAv2 | accuracy: 33.1 -> 55.0 | Improved |
| add reconstruction loss in Stage 2 LLM alignment (ablation) | X-Former · VQAv2 | accuracy: 55.0 -> 52.4 | Hurt |
| train X-Former in Stage 2 instead of keeping it frozen (ablation) | X-Former · GQA | accuracy: 25.5 -> 31.9 | Improved |
| replace X-Former with BLIP-2 style Q-Former on COCO captioning | X-Former · COCO captioning zero-shot | CIDEr: 131.0 -> 131.1 | No clear effect |
| add MAE branch via X-Former for object counting | BLIP-2 · Object Counting COCO | accuracy: 25.88 -> 39.64 | Improved |
| scale pretraining data to 105M | X-Former · GQA OPT-2.7B | accuracy: 34.1 -> 34.3 | Improved |
| add X-Former compared with BLIP-2 on Flickr30k image retrieval | BLIP-2 · Flickr30k IR@1 | IR@1: 83.5 -> 83.3 | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- BLIP-2 (14M) — BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (ICML 2022) · better on 13 of 15 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion (CVPR 2025 · similarity 0.89)
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation (NeurIPS 2025 · similarity 0.89)
- VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.89)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
