Research map / Vision-language and multimodal / Cross-modal learning
FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FreeBind, which augments a pre-trained audio-image-text unified space (ImageBind) with image-text and audio-text expert spaces via displacement and combination "space bonds" trained with lightweight projectors on pseudo-paired data.
Problem
Enhancing a pre-trained unified multimodal representation space (covering many modalities) with knowledge from expert single-pair spaces is costly and suffers from catastrophic forgetting when retraining.
Approach
Treat multimodal spaces as units and fuse them via two bonds: a displacement bond that remaps the unified space onto an expert space (inheriting expert knowledge), and a combination bond that remaps expert space onto a frozen unified space (preserving unified knowledge), composed sequentially and in parallel, plus a mixture-of-projectors and coarse-to-fine combining-factor inference.
Key results
- InternVL_IB (displacement bond) reaches 61.49 R@1 on COCO image-text retrieval versus 57.28 for ImageBind, and 81.54 ImageNet accuracy versus 76.31.
- ImageBind++ (AT Expertise setting) reaches 92.80 on ESC-50 and 25.35 mAP on AudioSet, improving over ImageBind's 67.25 and 13.96.
- InternVL_IB++ (AT E.) achieves 93.60 on ESC-50 and 20.19 R@1 on Clotho, surpassing the CLAP expert spaces on audio-text tasks in the reported settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Displacement-bond ImageBind with InternVL | ImageBind · COCO | R@1: 57.28 -> 61.49 | Helped |
| Add CLAP_m and CLAP_g using Audio-Text Expertise factors | ImageBind · AudioCaps | R@1: 40.25 -> 42.50 | Helped |
| Add CLAP_m and CLAP_g using Versatile factors | ImageBind · VGG-SS | R@1: 14.82 -> 15.48 | Helped |
| Use combination bond rather than displacement bond to integrate InternVL (ablation) | ImageBind · VGGSS | R@1: 13.77 -> 16.14 | Helped |
| Use displacement bond rather than combination bond to integrate InternVL (ablation) | ImageBind · COCO | R@1: 58.90 -> 61.49 | Helped |
| Use sequential displacement followed by parallel combination instead of either bond route alone (ablation) | ImageBind with InternVL, CLAP_m, and CLAP_g · COCO | R@1: 58.83 -> 61.13 | Helped |
| Use sequential displacement followed by parallel combination rather than displacement-only sequential integration (ablation) | ImageBind with InternVL, CLAP_m, and CLAP_g · AudioCaps | R@1: 36.03 -> 29.42 | Hurt |
| Add both CLAP_g and CLAP_m rather than either expert alone (ablation) | ImageBind · AudioCaps | R@1: 28.63 -> 29.16 | Helped |
| Ensemble all subset-trained projectors rather than using ψ_A alone (ablation) | ImageBind + Dr(InternVL) · AudioCaps | R@1: 9.59 -> 11.17 | Helped |
| Use the proposed projector pipeline instead of C-MCR's projector design (ablation) | ImageBind + Dr(InternVL) · AudioCaps | R@1: 10.43 -> 11.17 | Helped |
| Increase CLAP combination weights above 0.5 (ablation) | InternVL†_IB++ · Audio-text retrieval | R@1 | Helped |
| Set audio combining factor to a moderate value around 0.5 (ablation) | InternVL†_IB++ · Audio-image retrieval | R@1 | Helped |
Surprising findings
- Absorbing CLAP audio-text knowledge via combination bond also improves audio-image retrieval (e.g., VGG-SS R@1 rises from 14.82 to 29.16 range variants), although CLAP is trained only with audio-text alignment.
- Simpler single-projector training with only InfoNCE outperforms the more complex C-MCR and Ex-MCR projector pipelines in the reported ablations.
Limitations
- Only applied to the basic audio-image-text unified space; not yet extended to more modalities such as those in ImageBind or LanguageBind.
- Displacement bond degrades the original unified space's audio representations, trading off information for expert knowledge.
Experimental setup
Datasets: ImageNet AudioSet COCO CC3M MSRVTT MAD AudioCaps Clotho
Benchmarks: AudioCaps Clotho VGG-SS FlickrNet COCO Flickr30K ESC-50 AudioSet ImageNet
Metrics: R@1 R@5 accuracy mAP
Baselines: ImageBind WAV2CLIP AudioCLIP C-MCR Ex-MCR InternVL CLAP
Builds on: ImageBind C-MCR Ex-MCR InfoNCE
Closest papers by meaning
- OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces (ICLR 2025 · similarity 0.89)
- UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing (ICLR 2026 · similarity 0.84)
- Multimodal Representation Learning by Alternating Unimodal Adaptation (CVPR 2024 · similarity 0.84)
- Data-Efficient Multimodal Fusion on a Single GPU (CVPR 2024 · similarity 0.84)
- Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception (ICML 2026 · similarity 0.84)
- See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.