Research map / Vision-language and multimodal / Cross-modal learning
DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DH-Set, a set-embeddings vision-language alignment framework that dissects key local dimensions in each subspace for finer-grained diversity constraints and collapses the set into a single hybrid embedding at inference.
Problem
Vision-language alignment must handle semantic ambiguity (multiple meanings per image/text), but existing set-embeddings methods impose diversity constraints on whole subspaces and store/compute multiple embeddings, costing efficiency.
Approach
Learn sparse bottleneck vectors per subspace to identify key local dimensions carrying specific semantics, apply a KL-divergence diversity constraint on those dimensions across subspaces, share visual embeddings across subspaces, and fold per-subspace sparse weights into one hybrid embedding for O(d) inference.
Key results
- On Swin-384 + BERT, DH-Set reaches 559.4 rSum on Flickr30K and 551.3 on MS-COCO 5K, outperforming LAPS (544.1 on Flickr30K's comparable setting and 470.1 total).
- With ResNet152+BiGRU, DH-Set achieves 456.5 rSum on Flickr30K vs 442.6 for SET.
- Hybrid inference keeps accuracy equal to set similarity while reducing computation and storage from O(kd) to O(d), with k=3 reported as about 3x reduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DH-Set instead of SET | ResNet152 + BiGRU vision-language alignment · Flickr30K test | rSum: 442.6 -> 456.5 | Helped |
| Use DH-Set instead of LAPS | Swin-224 + BERT vision-language alignment · Flickr30K test | rSum: 536.3 -> 551.0 | Helped |
| Add each-subspace discriminability to all-subspace discriminability (ablation) | DH-Set · Flickr30K, Swin-224 + BERT | Image-to-Text R@1: 80.6 -> 83.7 | Helped |
| Use each-subspace discriminability only rather than all-subspace discriminability only (ablation) | DH-Set · Flickr30K, Swin-224 + BERT | Image-to-Text R@1: 80.6 -> 81.2 | Helped |
| Add semantic importance dissecting (ablation) | DH-Set · Flickr30K, Swin-224 + BERT | Image-to-Text R@1: 83.7 -> 85.2 | Helped |
| Add local diversity learning (ablation) | DH-Set · Flickr30K, Swin-224 + BERT | Image-to-Text R@1: 85.2 -> 88.3 | Helped |
| Use hybrid set-embeddings inference (ablation) | DH-Set · Flickr30K, Swin-224 + BERT | Image-to-Text R@1: 88.3 -> 88.3 | No clear effect |
| Increase set size from k=1 to k=3 (ablation) | DH-Set · Flickr30K test, Swin-224 + BERT | rSum: 545.4 -> 551.0 | Helped |
| Increase set size from k=3 to k=5 (ablation) | DH-Set · Flickr30K test, Swin-224 + BERT | rSum: 551.0 -> 549.5 | Hurt |
| Use visual sharing instead of textual sharing (ablation) | DH-Set hybrid inference · Flickr30K, Swin-224 + BERT | Image-to-Text R@1: 85.4 -> 88.3 | Helped |
| Use non-equivalent calculation in training and inference (ablation) | DH-Set hybrid inference · Flickr30K, Swin-224 + BERT | Image-to-Text R@1: 88.3 -> 58.1 | Hurt |
| Use only linear weights without transformation (ablation) | DH-Set hybrid inference · Flickr30K, Swin-224 + BERT | Image-to-Text R@1: 88.3 -> 82.0 | Hurt |
Surprising findings
- Unshared visual extractors achieve performance similar to shared ones, while textual sharing degrades results markedly.
- Learned sparse matrices reach roughly 5% non-zero dimensions.
Limitations
- Gains on the VLP backbones are smaller, with comparable rather than clearly superior performance to mainstream VLP models.
- Requires hyperparameters k and m to be tuned; performance gain saturates for k>=3.
Experimental setup
Datasets: Flickr30K MS-COCO
Benchmarks: Flickr30K MS-COCO 1K MS-COCO 5K
Metrics: R@1 R@5 R@10 rSum
Baselines: VSE++ PVSE PCME SET ESSE CHAN LG-MGC ESL DIAS NUIF SGR SCAN
Builds on: VSE++ PVSE PCME SET GPO
Closest papers by meaning
- Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.91)
- Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.90)
- Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models (ICML 2025 · similarity 0.89)
- VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.89)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.89)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.