Research map / Vision-language and multimodal / Human and embodied modeling
Asymmetric Visual Semantic Embedding Framework for Efficient Vision-Language Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVSE, an asymmetric image-text matching framework that samples multi-view ViT patch features via radial bias sampling and computes O(n) similarity by optimal matching of meta-semantic embeddings, with a dimension-wise regularization loss.
Problem
Image-text matching must bridge the information density gap between images (rich, multi-view) and text; global methods use static cosine similarity while local cross-attention methods are O(n^2) and slow at inference.
Approach
Asymmetric encoding: image embeddings from two radial-bias-sampled patch groups are compared to text via Asymmetric Embedding Optimal Matching, which splits embeddings into equal-dimension meta-semantic units and sums per-text-unit max cosine affinities, giving dynamic similarity with cosine-level cost; a cross-correlation regularizer aligns dimensions across views.
Key results
- On Flickr30K with ViT-Base-224 backbone, AVSE reaches 76.0 text-retrieval R@1 versus 74.0 for LAPS.
- With Swin-Base-384 on Flickr30K, AVSE reaches 87.1 text-retrieval R@1 versus 85.1 for LAPS.
- Ablation: AEOM with 2 views improves R@1 over cosine similarity with mean-pooled views (76.0 vs 74.0 text retrieval).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add AEOM instead of cosine similarity with 2 image views (ablation) | AVSE · Flickr30K, text retrieval | R@1: 74.0 -> 76.0 | Helped |
| Add AEOM instead of cosine similarity with 2 image views (ablation) | AVSE · Flickr30K, image retrieval | R@1: 59.9 -> 62.7 | Helped |
| Use 4 rather than 2 image views with AEOM (ablation) | AVSE · Flickr30K, text retrieval | R@1: 76.0 -> 75.9 | No clear effect |
| Set AEOM block dimension d2 to 256 rather than 64 (ablation) | AVSE · Flickr30K, text retrieval | R@1: 74.5 -> 76.0 | Helped |
| Set AEOM block dimension d2 to 256 rather than 128 (ablation) | AVSE · Flickr30K, text retrieval | R@1: 75.8 -> 76.0 | No clear effect |
| Replace uniform sampling with radial bias sampling (ablation) | AVSE · Flickr30K, text retrieval | R@1: 73.4 -> 76.0 | Helped |
| Replace Gaussian sampling with radial bias sampling (ablation) | AVSE · Flickr30K, text retrieval | R@1: 74.9 -> 76.0 | Helped |
| Add dimension-wise regularizing loss (ablation) | AVSE · Flickr30K, text retrieval | R@1: 74.8 -> 76.0 | Helped |
| Add dimension-wise regularizing loss (ablation) | AVSE · Flickr30K, image retrieval | R@1: 61.8 -> 62.7 | Helped |
| Use AVSE rather than LAPS | ViT-Base-224 image-text retrieval · Flickr30K 1K Test, text retrieval | R@1: 74.0 -> 76.0 | Helped |
Surprising findings
- Using 4 views does not improve over 2 views, which the authors attribute to redundant information across views.
Limitations
- Gains over LAPS are modest on several metrics and backbones.
- Increasing the number of views beyond two gave no further benefit.
Experimental setup
Datasets: MS-COCO Flickr30K
Benchmarks: Flickr30K 1K test MS-COCO 5-fold 1K test MS-COCO 5K test
Metrics: Recall@1 Recall@5 Recall@10
Baselines: VSE++ SCAN SGR CHAN LAPS NAAF HREM
Builds on: VSE++ SCAN LAPS BERT
Closest papers by meaning
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.88)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.87)
- Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models (ICML 2025 · similarity 0.87)
- Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.87)
- Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere (NeurIPS 2025 · similarity 0.86)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.