Research map / Vision-language and multimodal / Cross-modal learning
TextME: Bridging Unseen Modalities Through Text Descriptions
Soyeon Hong, Jinchan Kim, Jaegook You, Seungtaek Choi, Suha Kwak, Hyunsouk Cho
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add offset (centroid) correction (ablation) | TextME · ModelNet40 | Top-1: 4.05 -> 70.86 | Improved |
| add offset (centroid) correction (ablation) | TextME · AudioCaps | R@1: 8.68 -> 15.35 | Improved |
| add offset correction on molecules (gap consistency 0.78) (ablation) | TextME · DrugBank | MRR: 36.44 -> 34.75 | Hurt |
| train projection on domain captions instead of general all-NLI text (ablation) | TextME · ModelNet40 | Top-1: 12.10 -> 70.86 | Improved |
| train projection on domain captions instead of general all-NLI text (ablation) | TextME · DrugBank | MRR: 16.10 -> 34.75 | Improved |
| use LLM embedding anchor instead of multimodal encoder anchor (ablation) | TextME · AudioCaps | R@1: 14.5 -> 16.2 (NV-Embed-v2 vs LanguageBind) | Improved |
| use LLM embedding anchor instead of multimodal encoder anchor (ablation) | TextME · ESC-50 | Top-1: 86.7 -> 77.2 (CLIP vs Qwen3-Embed) | Hurt |
| ensemble multiple anchors by score-level late fusion (ablation) | TextME · AudioCaps | R@1: 15.4 -> 19.2 | Improved |
| ensemble multiple anchors by score-level late fusion (ablation) | TextME · RSNA | Top-1: 46.6 -> 46.9 | No clear effect |
| lower variance of gap-content orthogonality in the encoder | TextME · all tasks (33 data points) | performance preservation: Pearson r = -0.67, p < 0.001 | Improved |
| use TextME instead of unpaired COX baseline | TextME · COCO retrieval | recall: 0.02 -> 28.63 | Improved |
Closest papers by meaning
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.90)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.89)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.88)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.88)
- Gramian Multimodal Representation Learning and Alignment (ICLR 2025 · similarity 0.88)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
