Research map / Vision-language and multimodal / Multimodal LLMs

Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE

ICLR 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Octavius, an MLLM whose LLM decoder uses LoRA-MoE with instance-based (question-conditioned) gating over LoRA experts, plus a Object-As-Scene 3D point cloud encoder, to mitigate task interference when training on 2D and 3D instruction data.

Problem

Simultaneously instruction-tuning an MLLM on multiple tasks and modalities (2D detection, captioning, VQA; 3D captioning, VQA, classification) causes task interference ("tug-of-war") that degrades performance under PEFT.

Approach

Replace a single LoRA in each projection of the LLM with a set of LoRA experts routed by a sparse top-k gate computed from the instance's question (instance-based gate rather than token-based gate), so different tasks use different experts; combined with a scene-level 3D encoder that aggregates Point-BERT RoI features via query-based cross-attention.

Key results

• LoRA-MoE raises PASCAL VOC detection recall@0.5 from 7.61 to 39.04 and ScienceQA-style VQA accuracy from 40.31 to 46.95 on the 2D setup.
• On joint 2D+3D training, MoE improves average performance by about 21.4% (detection recall 2.64 to 34.30; ScanNet cls 38.80 to 47.20).
• Object-As-Scene with MoE reaches ScanQA CIDEr 167.31 and Nr3d zero-shot CIDEr 17.22, outperforming 3D-LLM on ScanQA (59.20 CIDEr).

Measured effects reported in the paper

Surprising findings

• Load-balancing loss does not improve results and is described as incompatible with instance-based gating, so expert usage remains imbalanced (e.g., only 3 of 4 experts used).
• Adding a system prompt to the gate input degrades performance, attributed to redundant global information.

Limitations

• Joint multimodal training still degrades performance relative to single-modality training, with a gap remaining in several tasks.
• Load balancing across experts is not solved; token-based gating with balancing is left for future work.
• Many ablations use single runs on a limited set of tasks and benchmarks; the paper's own pilot study uses a single LAMM setup.

Experimental setup

Datasets: MS-COCO Bamboo LAMM v2 ScanNet Scan2Inst PASCAL VOC ScienceQA LLaVA instruction data ShapeNet NR3D Flickr30K CIFAR-10

Benchmarks: PASCAL VOC ScienceQA Flickr30K CIFAR-10 CelebA ShapeNet NR3D Scan2Cap ScanQA ScanNet

Metrics: Recall@0.5 Precision@0.5 Acc@1 CIDEr BLEU-1 BLEU-4 METEOR ROUGE-L

Baselines: LAMM LLaVA-LoRA 3D-LLM (Flamingo) Ours w/o MoE

Builds on: LAMM LLaVA LoRA Mixture-of-Experts ULIP Point-BERT 3D-LLM

Closest papers by meaning

• Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning (ICML 2025 · similarity 0.87)
• LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
• MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs (ICLR 2025 · similarity 0.87)
• MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models (NeurIPS 2024 · similarity 0.87)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.86)
• Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.86)