Research map / Vision-language and multimodal / Multimodal LLMs
Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Omni-SMoLA, which softly mixes many low-rank experts (Soft MoE over LoRA-style adapters) for text, visual, and multimodal tokens on top of PaLI models, to improve generalist LMM performance without the parameter cost of conventional MoE.
Problem
Generalist large multimodal models degrade when instruction-tuned on many diverse vision-language tasks, while conventional MoE experts at LMM scale are too costly to replicate, limiting the number of experts.
Approach
Attach Soft MoE routing to zero-initialized low-rank (rank 4) LoRA-style experts on linear layers of the PaLI encoder, with three expert sets (visual-token, text-token, multimodal-token) that residually specialize on a frozen-in-spirit large backbone, allowing hundreds of cheap experts.
Key results
- Omni-SMoLA on PaLI-X (48 experts per modality, FT) reaches a COCO Karpathy CIDEr of 149.8, a new specialist-beating generalist SOTA, and averages +0.38 over PaLI-X specialists across the captioning/VQA benchmarks in the ablation table.
- With the text-heavy OCR-input setting, the PaLI-X-based generalist sets new SOTA on TextVQA, ST-VQA, InfoVQA, DocVQA, AI2D and ChartQA, outperforming specialist SOTA on 6 of 10 datasets.
- Inference speed drops only 1.4% (31.29 to 30.85 examples/s) when adding 48 experts per SMoLA block on PaLI-3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 48 experts per modality to PaLI-X FT | PaLI-X FT · COCO Karpathy-test | CIDEr: 148.7 -> 149.8 | Helped |
| add 48 experts per modality to PaLI-X FT | PaLI-X FT · ChartQA test, with OCR | accuracy: 70.2 -> 73.8 | Helped |
| add 48 experts per modality to PaLI-3 FT | PaLI-3 FT · VQAv2 test-dev | accuracy: 82.5 -> 83.6 | Helped |
| switch from PaLI-X FT to PaLI-X LoRA as the SMoLA starting checkpoint (ablation) | Omni-SMoLA · full-mixture benchmark average | average score gain: +1.26 vs +1.64 | Mixed |
| use a raw PaLI-3 checkpoint instead of the fine-tuned checkpoint (ablation) | Omni-SMoLA · full-mixture benchmarks | average score | Helped |
| use modality-specific visual, text, and multimodal experts rather than experts over all tokens (ablation) | Omni-SMoLA PaLI-3 FT · full-mixture validation tasks | average score | Helped |
| increase expert count from 4 to 16 per modality (ablation) | Omni-SMoLA PaLI-3 FT · full-mixture validation tasks | average score gain: 1.1 -> 1.14 | Helped |
| increase expert count from 16 to 48 per modality (ablation) | Omni-SMoLA PaLI-3 FT · full-mixture validation tasks | average score gain: 1.14 -> 1.2 | Helped |
| increase expert count from 48 to 144 per modality (ablation) | Omni-SMoLA PaLI-3 FT · full-mixture validation tasks | average score gain: 1.2 -> 1.27 | Helped |
| replace SMoLA experts with higher-rank LoRA (ablation) | PaLI-3 FT · full-mixture validation tasks | average score gain: 0.86 -> 0.79 -> 0.59 | Hurt |
| add per-task LoRA tuning to Omni-SMoLA (ablation) | Omni-SMoLA PaLI-X FT · COCO Karpathy-test | CIDEr | Helped |
| add 48 experts per modality to PaLI-3 FT (ablation) | PaLI-3 FT with VQA mixture · OKVQA | accuracy: a 0.2 performance drop | Hurt |
Surprising findings
- Higher-rank LoRA (rank 384 or 1536) did not improve over lower rank, and the effective rank of high-rank LoRA weights was small, while SMoLA with many low-rank experts kept improving with expert count.
Limitations
- Gains over specialists are small on average (e.g., +0.38 for PaLI-X FT, +0.64 for PaLI-3 captioning), and generalist still trails specialists on several tasks such as TextCaps.
- Experiments use proprietary PaLI backbones (5B and 55B) and no open-weight models, limiting reproducibility.
- Expert count scaling gains saturate quickly (1.1 to 1.27 points from 4 to 144 experts).
Experimental setup
Datasets: COCO captions TextCaps VizWiz-Cap VQAv2 OK-VQA VizWiz-VQA ST-VQA TextVQA OCRVQA InfoVQA DocVQA ChartQA
Benchmarks: COCO Captions NoCaps VQAv2 OKVQA A-OKVQA ScienceQA TallyQA TextCaps VizWiz-Cap TextVQA VizWiz-VQA ST-VQA
Metrics: CIDEr VQA accuracy average delta vs specialist inference examples per second
Baselines: PaLI-3 specialist PaLI-X specialist PaLI-3 FT (generalist) PaLI-X FT (generalist) PaLI-X LoRA GIT2 BEiT-3 PaLM-E InstructBLIP CogVLM Qwen-VL Unified-IO
Builds on: Soft MoE LoRA PaLI-X PaLI-3
Closest papers by meaning
- On the generalization capacity of neural networks during generic multimodal reasoning (ICLR 2024 · similarity 0.87)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.87)
- LLaVA-Critic: Learning to Evaluate Multimodal Models (CVPR 2025 · similarity 0.87)
- Re-Imagining Multimodal Instruction Tuning: A Representation View (ICLR 2025 · similarity 0.86)
- R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts (ICML 2025 · similarity 0.86)
- RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.