Research map / Vision-language and multimodal / Multimodal LLMs
OneLLM: One Framework to Align All Modalities with Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents OneLLM, an MLLM that aligns eight modalities (image, video, audio, point cloud, depth/normal maps, IMU, fMRI) to LLaMA2-7B through a shared frozen CLIP-ViT encoder, a soft-routed universal projection module of mixed image-pretrained experts, and progressive alignment plus a 2M-item instruction dataset.
Problem
Existing multimodal LLMs rely on modality-specific encoders and projections that differ in architecture and are limited to common modalities such as image, audio, and video, making it hard to scale to many modalities within one framework.
Approach
Use one frozen vision-language encoder (CLIP-ViT) shared across all modalities, with lightweight per-modality convolutional tokenizers, learnable modality tokens, and a Universal Projection Module that mixes several image-pretrained projection experts via a soft router; modalities are added progressively by data scale.
Key results
- OneLLM-7B reaches 71.6 VQAv2 and 115.9 NoCaps CIDEr, outperforming AnyMAL-13B and ChatBridge-13B on most image-text benchmarks.
- On zero-shot video QA, OneLLM-7B scores 57.3 on NextQA and 65.7 on How2QA, above InternVideo and FrozenBiLM.
- On Clotho AQA, OneLLM-7B reaches 57.9 accuracy, on par with fully finetuned Pengi (64.5 is Pengi's reported figure, so it is close but not equal).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly train modalities rather than train each modality separately (ablation) | OneLLM · ClothoQA | score: 37.8 -> 57.4 | Helped |
| jointly train modalities rather than train each modality separately (ablation) | OneLLM · MSVDQA | score: 31.0 -> 56.8 | Helped |
| jointly train modalities rather than train each modality separately (ablation) | OneLLM · NoCaps and VQAv2 | score: NoCaps: 115.6 -> 115.8; VQAv2: 71.9 -> 71.6 | No clear effect |
| initialize projection experts from image-text alignment instead of random initialization (ablation) | OneLLM · NoCaps | CIDEr: 98.8 -> 115.8 | Helped |
| initialize projection experts from image-text alignment instead of random initialization (ablation) | OneLLM · VQAv2 | accuracy: 65.6 -> 71.6 | Helped |
| use 3 projection experts rather than 1 (ablation) | OneLLM UPM · NoCaps | CIDEr: 108.7 -> 115.8 | Helped |
| use 5 or 7 projection experts rather than 3 (ablation) | OneLLM UPM · NoCaps | CIDEr: 3: 115.8; 5: 114.6; 7: 114.9 | No clear effect |
| use a soft router rather than a constant router (ablation) | OneLLM UPM · NoCaps | CIDEr: 109.8 -> 115.8 | Helped |
| use a soft router rather than a sparse router (ablation) | OneLLM UPM · NoCaps | CIDEr: 112.8 -> 115.8 | Helped |
| unfreeze CLIP during multimodal-text alignment (ablation) | OneLLM · NoCaps | CIDEr: 115.8 -> 106.0 | Hurt |
| unfreeze CLIP during multimodal-text alignment (ablation) | OneLLM · ClothoQA | score: 57.4 -> 62.1 | Helped |
| replace frozen CLIP with frozen DINOv2 as universal encoder (ablation) | OneLLM · NoCaps | CIDEr: 115.8 -> 104.6 | Hurt |
Surprising findings
- Joint training across modalities substantially helps data-scarce modalities: separately trained audio and video models are much worse on ClothoQA (37.8 vs 57.4) and MSVDQA (31.0 vs 56.8).
- Trainable CLIP encoder improves audio QA by 4.7 points but hurts visual tasks, so the frozen encoder is kept for memory and performance balance.
Limitations
- Gap to specialized models on non-image modalities due to limited high-quality data.
- Weak fine-grained understanding of high-resolution images, long videos, and long audio.
Experimental setup
Datasets: LAION-400M LAION-COCO WebVid-2.5M WavCaps Cap3D CC3M Ego4D NSD LLaVA-150K COCO Caption VQAv2 GQA
Benchmarks: GQA VQAv2 OKVQA TextVQA ScienceQA VizWiz NoCaps Flickr30K MME MMBench MMVet SEED-Bench
Metrics: accuracy CIDEr ROUGE-L BLEU-1 METEOR SPIDEr
Baselines: X-LLM ChatBridge AnyMAL ImageBind-LLM PandaGPT PointLLM InstructBLIP Qwen-VL LLaVA-v1.5 BLIP-2 Flamingo
Builds on: CLIP LLaMA2 LLaMA-Adapter Soft MoE ImageBind Meta-Transformer
Closest papers by meaning
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.90)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.90)
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.89)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.89)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.