Research map / Vision-language and multimodal / Human and embodied modeling
VIRTUE: Visual-Interactive Text-Image Universal Embedder
Wei-Yao Wang, Kazuya Tateishi, Qiyu Wu, Shusuke Takahashi, Yuki Mitsufuji
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SAM-2 segmentation embeddings (VIRTUE) to a VLM embedder with same data | Qwen2-VL-2B VLM2Vec · MMEB overall | precision@1: 59.7 -> 64.8 | Improved |
| add segmentation embeddings (VIRTUE) | 7B embedders · MMEB overall | precision@1: 66.6 -> 68.6 | Improved |
| add segmentation embeddings (VIRTUE) | VLM2Vec-2B · SCaR overall | precision@1: 24.1 -> 30.4 | Improved |
| add segmentation embeddings (VIRTUE) | 7B embedders · SCaR overall | precision@1: 26.3 -> 27.8 | Improved |
| fine-tune on SCaR-train | VIRTUE-2B · SCaR overall | precision@1: 30.4 -> 56.2 | Improved |
| crop the region of interest instead of using visual prompt (ablation) | VIRTUE-2B and other models · SCaR overall | precision@1: 30.4 -> 24.8 | Hurt |
| add red circle visual hint to CLIP-based ReCLIP | ReCLIP · SCaR overall | precision@1: 18.1 -> 19.0 | No clear effect |
| remove task instruction from query (ablation) | VIRTUE-2B · MMEB overall | precision@1: 59.2 -> 64.8 | Improved |
| use SAM-2 instead of Prompter or cropping (ablation) | VIRTUE-2B · SCaR overall | precision@1: 22.7 (Prompter), 25.9 (Cropped) -> 30.4 | Improved |
| increase SAM2.1 size from B+ to L (ablation) | VIRTUE-2B · SCaR overall | precision@1: 30.4 -> 27.9 | Hurt |
| increase segmentation embedding length from 256 to 1024 (ablation) | VIRTUE-2B · MMEB vs SCaR | precision@1: MMEB 64.8 -> 60.2; SCaR 30.4 -> 35.1 | Mixed |
| unfreeze SAM-2 during fine-tuning (ablation) | VIRTUE-2B · SCaR overall | precision@1: 30.4 -> 25.8 | Hurt |
Later papers that compared with this method
1 later accepted paper reports VIRTUE-2B as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Omni-Interactive Universal Embedder (NeurIPS 2026 · better on 2 of 2 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.89)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.88)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.87)
- From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.86)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.86)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
