Research map / Vision-language and multimodal / Cross-modal learning
Achieving Ensemble-Like Performance in a Single Model: A Feature Diversification Framework for Image-Text Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Feature Diversification Framework (FDF) that turns shared image/text features into multiple realigned, reweighted feature conversions to get ensemble-like image-text matching performance within a single model.
Problem
Model ensembling improves image-text matching but requires training and running multiple models, which is inefficient; the paper asks whether a single model can achieve ensemble-like performance.
Approach
Cross-model realignment shows that feature extractors from differently initialized models yield similar features, so diversified features can be derived from one shared extractor by lightweight feature conversions (FFN heads), trained with a realignment strategy that uses previous minibatches as diverse negatives, and fused via learned reweighting.
Key results
- On Flickr30K, CHAN+FDF with BERT reaches 530.8 RSUM versus 518.6 for the CHAN baseline, a gain of 12.2 RSUM.
- On MS-COCO 5-fold 5K with BiGRU, CHAN+FDF improves RSUM from 432.7 to 441.4 (+8.7).
- With 2 feature conversions, FDF matches an ensemble of 2 CHAN models at about 10% of the cost of a new model, with inference time increase of about 4.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add FDF to CHAN | CHAN with BUTD + BiGRU · MS-COCO 5-fold 1K Test | RSUM: 525.3 -> 529.3 (+4.0) | Helped |
| Add FDF to CHAN | CHAN with BUTD + BiGRU · MS-COCO 5-fold 5K Test | RSUM: 432.7 -> 441.4 (+8.7) | Helped |
| Add FDF to CHAN | CHAN with BUTD + BERT · MS-COCO 5-fold 1K Test | RSUM: 531.3 -> 533.3 (+2.0) | Helped |
| Add FDF to CHAN | CHAN with BUTD + BERT · MS-COCO 5-fold 5K Test | RSUM: 441.9 -> 447.6 (+5.7) | Helped |
| Add FDF to CHAN | CHAN with BERT · Flickr30K | RSUM: 518.6 -> 530.8 (+12.2) | Helped |
| Add feature conversion (FC) (ablation) | CHAN with BERT · Flickr30K | RSUM: 518.6 -> 521.5 | Helped |
| Add realignment training strategy (RTS) (ablation) | CHAN with BERT and feature conversion · Flickr30K | RSUM: 521.5 -> 527.0 | Helped |
| Add reweighting module (RW) (ablation) | CHAN with BERT and realignment training · Flickr30K | RSUM: 527.0 -> 528.5 | Helped |
| Use no additional structure in feature conversion (ablation) | FDF with CHAN and BERT · Flickr30K | RSUM: 518.6 -> 523.6 | Helped |
| Use adapter structure for feature conversion (ablation) | FDF with CHAN and BERT · Flickr30K | RSUM: 518.6 -> 525.6 | Helped |
| Use FFN structure for feature conversion (ablation) | FDF with CHAN and BERT · Flickr30K | RSUM: 518.6 -> 528.5 | Helped |
Surprising findings
- Feature extractors from differently initialized models produce similar features, so features can be transformed by fine-tuning only a small set of parameters.
- Negative sample differences across initializations drive feature diversification, and realignment training with previous minibatches as negatives improved RSUM by 8.4% over baseline.
Limitations
- Training duration increases (about 14%) because multiple feature conversions must be optimized concurrently.
- Gains are reported mainly for retrieval on two datasets; results depend on the specific baseline architectures.
Experimental setup
Datasets: MS-COCO Flickr30K
Benchmarks: MS-COCO 5-fold 1K MS-COCO 5K Flickr30K
Metrics: R@1 R@5 R@10 RSUM
Baselines: SCAN CHAN VSE∞ SGRAF CGMN NAAF DivE HREM TERAN USER DCIN model ensembling (2 models)
Builds on: VSE∞ CHAN SCAN VSE++ bottom-up top-down attention
Closest papers by meaning
- Expanding the Scope of Negatives: Boosting Image-Text Matching with Negatives Distribution Guided Learning (AAAI 2025 · similarity 0.86)
- MATCHA: Towards Matching Anything (CVPR 2025 · similarity 0.84)
- Information Theoretic Text-to-Image Alignment (ICLR 2025 · similarity 0.84)
- ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.84)
- Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.84)
- Improving Long-Text Alignment for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.