Research map / Vision-language and multimodal / Cross-modal learning
UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniBind, which builds an LLM- and MLLM-generated text knowledge base, forms top-50 text embedding centers per class, and aligns frozen CLIP-style multi-modal encoders (plus linear layers) to these modality-agnostic centers for a balanced 7-modality space.
Problem
Existing unified multi-modal spaces such as ImageBind use images as the central alignment modality, producing unbalanced representations, and category-name text embeddings are too weak to represent visual semantics.
Approach
Replace the image-centered alignment with modality-agnostic, LLM-augmented text embedding centers: a knowledge base of LLM category descriptions and multi-modal LLM image/event/point-cloud descriptions, from which the top-50 text embeddings per class serve as class centers for contrastive alignment and max-similarity recognition.
Key results
- Zero-shot gains over the base models, averaging about +6.27% top-1 accuracy across benchmarks; e.g., +5.55% on ImageNet-1K with ImageBind and +8.28% on N-Caltech with PointBind.
- Fine-tuning with a linear layer reaches 86.94% on ImageNet-1K (+6.75% over PointBind w/ linear) and 56.99% on Places-365, while training only about 10% of learnable parameters.
- Event-to-image retrieval R@20 on PointBind improves from 37.70 to 55.66 (+17.96%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LLM-augmented contrastive learning (ablation) | PointBind · Event-to-Image retrieval | R@20: 37.70 -> 55.66 (+17.96) | Helped |
| add LLM-augmented contrastive learning (ablation) | PointBind · Image-to-Event retrieval | R@1: 14.07 -> 14.12 (+0.05) | No clear effect |
| add LLM-augmented contrastive learning (ablation) | E-CLIP · Image-to-Event retrieval | R@1: 79.52 -> 78.95 (-0.57) | Hurt |
| add LLM-augmented contrastive learning (ablation) | E-CLIP · Event-to-Image retrieval | R@20: 94.61 -> 96.70 (+2.09) | Helped |
| use localized embedding centers instead of category-name prompt centers (ablation) | E-CLIP · ImageNet-1K | accuracy: 68.30 -> 78.63 (+10.33) | Helped |
| use localized embedding centers instead of category-name prompt centers (ablation) | PointBind · N-Caltech-101 | accuracy: 50.98 -> 59.26 (+8.28) | Helped |
| use localized embedding centers instead of category-name prompt centers (ablation) | AudioCLIP · ESC-50 | accuracy: 68.60 -> 71.25 (+2.65) | Helped |
| use localized embedding centers instead of category-name prompt centers (ablation) | PointBind · ModelNet40 | accuracy: 77.67 -> 80.59 (+2.92) | Helped |
| build the knowledge base with both LLM and multimodal-LLM descriptions rather than either source alone (ablation) | UniBind · zero-shot recognition | accuracy | Helped |
| select the top 50 descriptions per category for embedding-center localization (ablation) | UniBind · zero-shot recognition | accuracy | Helped |
| use LLM-generated prompts rather than the proposed localized embedding centers (ablation) | UniBind · ImageNet-1K | accuracy: 79.59 -> 83.25 | Hurt |
Surprising findings
- Embedding centers from top-50 LLM descriptions gave gains larger than LLM-generated prompts alone, and the authors found that using both LLM and MLLM text sources performed best.
Limitations
- Robustness of the LLM-augmented method needs improvement, as acknowledged by the authors.
- Relies on the quality of LLM and multi-modal LLM generated descriptions.
- Gains on some benchmarks are small or negative (e.g., MSR-VTT fine-tuning and N-Caltech retrieval R@1 for E-CLIP).
Experimental setup
Datasets: ImageNet-1K Places-Stanford-365 Caltech-101 ModelNet-40 ShapeNet-part ESC-50 Urban-Sound-8K LLVIP RGB-T MSR-VTT UCF-101 N-Caltech-101
Benchmarks: ImageNet-1K Places-365 ModelNet40 ShapeNet ESC-50 Urban-Sound-8K LLVIP RGB-T MSR-VTT UCF-101 N-Caltech-101 N-ImageNet-1K
Metrics: top-1 accuracy recall@k
Baselines: ImageBind PointBind CLIP E-CLIP AudioCLIP PointCLIP Meta-Transformer LLM-generated prompts (PointCLIP V2)
Builds on: ImageBind PointBind CLIP E-CLIP AudioCLIP PointCLIP
Closest papers by meaning
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.86)
- UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.86)
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.85)
- Gramian Multimodal Representation Learning and Alignment (ICLR 2025 · similarity 0.85)
- Omni-Interactive Universal Embedder (NeurIPS 2026 · similarity 0.85)
- CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.