Research map / Vision-language and multimodal / Cross-modal learning
CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CrossGET, a training-light token ensemble framework that uses learnable cross tokens for bidirectional cross-modal guidance and a parallel complete-graph soft matching algorithm to accelerate CLIP, BLIP, BLIP2 and LLaVA-1.5 vision-language Transformers.
Problem
Vision-language Transformers are computationally expensive; token reduction has been studied for unimodal models but cross-modal guidance is hard to use in modality-dependent models (e.g., BLIP, BLIP2, LLaVA) where the vision branch runs before the language branch.
Approach
Inject learnable cross tokens into each modality, trained with a JS-divergence alignment loss on detached projection layers, so each modality can receive cross-modal importance before the other modality runs; combine this with a non-iterative, parallelizable complete-graph soft matching that merges tokens via softmax-weighted ensembling.
Key results
- On CLIP Flickr30K retrieval, CrossGET keeps image-to-text R@1 at 92.1 and raises text-to-image R@1 to 79.6 (vs 79.3) while cutting GFLOPs by 42% and raising throughput by 57%.
- On BLIP NLVR2, CrossGET gets 83.2 test accuracy (0.2 below BLIP's 83.4) while saving 57% GFLOPs and improving throughput by 93%.
- On LLaVA-1.5-7B with 10% SFT data, CrossGET keeps more than 98% of average performance while cutting forward GFLOPs by 31% and raising forward throughput by 89%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CrossGET token ensembling | CLIP · Flickr30K Karpathy split | Average Recall@1: 85.7 -> 85.9 | Helped |
| Add cross-guided matching to complete-graph soft matching (ablation) | CrossGET CGSM · Flickr30K Karpathy split | Average Recall@1: 85.0 -> 85.8 | Helped |
| Add cross-guided ensemble to cross-guided matching (ablation) | CrossGET CGSM+CGM · Flickr30K Karpathy split | Average Recall@1: 85.8 -> 85.9 | Helped |
| Add CrossGET token ensembling | BLIP · NLVR2 test | accuracy: 83.4 -> 83.2 | Hurt |
| Add CrossGET token ensembling | BLIP · COCO Caption | CIDEr: 133.3 -> 131.6 | Hurt |
| Add CrossGET token ensembling | BLIP · NoCaps entire | CIDEr: 109.9 -> 108.1 | Mixed |
| Add CrossGET token ensembling | BLIP · VQA2.0 test-dev | overall accuracy: 77.4 -> 77.0 | Hurt |
| Add CrossGET token ensembling with tuning | BLIP2-OPT6.7B · COCO Caption | CIDEr: 144.5 -> 143.1 | Hurt |
| Apply CrossGET to both vision and language modalities instead of vision only (ablation) | CLIP · Flickr30K Karpathy split | Image-to-text Recall@1: 92.1 -> 91.4 | Hurt |
| Use one cross token in the first layer instead of separate tokens in each layer (ablation) | CrossGET · Flickr30K Karpathy split | Image-to-text Recall@1: 91.5 vs. 92.1 | Hurt |
| Replace informative cross-token initialization with zero initialization (ablation) | CrossGET · Flickr30K Karpathy split | Text-to-image Recall@1: 77.9 vs. 79.6 | Hurt |
| Replace informative cross-token initialization with normal-random initialization (ablation) | CrossGET · Flickr30K Karpathy split | Image-to-text Recall@1: 90.4 vs. 92.1 | Hurt |
Surprising findings
- Simply adding an extra learnable token to ToMe does not notably improve results, suggesting the gains come from cross-modal information rather than extra tokens.
- Random or zero initialization of cross tokens performs noticeably worse than initializing from informative [CLS]/[BOS] tokens in fine-tuning.
Limitations
- Cross tokens require training (fine-tuning) to reach best performance; without training accuracy drops.
- Gains on few-shot CoOp classification are smaller (about 1.8 points loss at 42% savings), attributed to frozen backbone and overfitting.
- Extension to interleaved vision-language inputs is only discussed, not evaluated.
Experimental setup
Datasets: Flickr30K COCO Caption NLVR2 VQAv2 NoCaps LLaVA-1.5 SFT data (10% sample) ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101
Benchmarks: Flickr30K NLVR2 COCO Caption NoCaps VQAv2 VQAv2 (LLaVA eval) GQA VisWiz ScienceQA-IMG TextVQA POPE MME
Metrics: R@1 CIDEr SPICE BLEU@4 accuracy GFLOPs throughput
Baselines: ToMe UPop TRIPS DynamicViT EViT Hourglass CoOp
Builds on: ToMe BLIP BLIP2 CLIP CoOp
Closest papers by meaning
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval (AAAI 2025 · similarity 0.88)
- CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer (AAAI 2024 · similarity 0.88)
- DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs (NeurIPS 2024 · similarity 0.87)
- Enhancing Multi-Image Understanding through Delimiter Token Scaling (ICLR 2026 · similarity 0.87)
- TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.