Research map / Vision-language and multimodal / Human and embodied modeling
Cross Modal Fine-grained Alignment via Granularity-aware and Region-uncertain Modeling
Jiale Liu, Haoming Zhou, Yishu Liu, Bingzhi Chen, Yuncheng Jiang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Significance-aware Adapter (ablation) | GRM · Flickr30K 1k (ViT-Base-224) | rSum: 502.8 -> 516.2 | Improved |
| add Granularity-aware Adapter (ablation) | GRM · Flickr30K 1k (ViT-Base-224) | rSum: 507.0 -> 516.2 | Improved |
| add region prompts (ablation) | GRM · Flickr30K 1k (ViT-Base-224) | rSum: 503.3 -> 516.2 | Improved |
| add region-level uncertainty modeling (ablation) | GRM · Flickr30K 1k (ViT-Base-224) | rSum: 508.3 -> 516.2 | Improved |
| add original-feature contrastive loss (ablation) | GRM · Flickr30K 1k (ViT-Base-224) | rSum: 487.2 -> 516.2 | Improved |
| add significance-aware contrastive loss (ablation) | GRM · Flickr30K 1k (ViT-Base-224) | rSum: 505.9 -> 516.2 | Improved |
| add uncertainty-aware contrastive loss (ablation) | GRM · Flickr30K 1k (ViT-Base-224) | rSum: 501.6 -> 516.2 | Improved |
| add semantic consistency (reconstruction) loss (ablation) | GRM · Flickr30K 1k (ViT-Base-224) | rSum: 513.2 -> 516.2 | Improved |
| add KL and entropy regularization (ablation) | GRM · Flickr30K 1k (ViT-Base-224) | rSum: 514.2 -> 516.2 | Improved |
| increase number of region prompts beyond 5 (ablation) | GRM with ViT-Base-224 · Flickr30K | rSum | Hurt |
| use 50 region prompts with Swin-Base-224 backbone (ablation) | GRM with Swin-Base-224 · Flickr30K | rSum | Improved |
| put heavy weight c=0.6 on uncertainty-aware alignment loss (a=b=0.2) (ablation) | GRM · Flickr30K | rSum | Hurt |
Closest papers by meaning
- Linguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal Alignment (CVPR 2024 · similarity 0.91)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.90)
- Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.90)
- Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models (NeurIPS 2025 · similarity 0.89)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.89)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
