Research map / Architectures and efficiency / Mixture of experts
Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STGC, a plug-in that uses token-level gradients to flag conflicting tokens inside MoE experts and adds a conflict elimination loss to re-route them, improving MoE-LLaVA on LVLM benchmarks.
Problem
Routers in MoE-based large vision-language models are not optimized for parameter-gradient conflicts between tokens assigned to the same expert, causing interference and sub-optimal expert learning.
Approach
Identify conflicting tokens as those whose per-token gradient on an expert has negative cosine similarity to the expert's average gradient, then add a cross-entropy-style conflict elimination loss that pushes their router scores away from the current expert, reducing gradient conflicts without updating parameters during identification.
Key results
- With STGC, MoE-LLaVA (Phi-2 2.7B, 4Top2, 3.6B activated) improves average from 61.1 to 61.8 across benchmarks, and the stronger-encoder variant from 62.9 to 63.5.
- The full method with siglip encoder and 1021K data reaches 61.0 average at 3.6B activated, with ChartQA 44.7 and DocVQA 42.1 reported.
- Adding STGC to MoE-LLaVA with StableLM-1.6B (4Top2) improves average from 57.3 to 58.0 and outperforms embedding-, feature-, and task-based routing baselines (56.9, 57.2, 48.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add token-gradient conflict elimination to MoE-LLaVA | MoE-LLaVA, Phi-2-2.7B, 4Top2, siglip visual encoder · VisWiz | accuracy: 43.7 -> 48.6 | Helped |
| Add token-gradient conflict elimination to MoE-LLaVA | MoE-LLaVA, Phi-2-2.7B, 4Top2, siglip visual encoder · MMBench | accuracy: 68.0 -> 71.0 | Helped |
| Add token-gradient conflict elimination to MoE-LLaVA | MoE-LLaVA, Phi-2-2.7B, 4Top2, siglip visual encoder · MM-Vet | score: 35.9 -> 40.7 | Helped |
| Add STGC to MoE-LLaVA-4Top2 (ablation) | MoE-LLaVA, StableLM-1.6B · TextVQA | accuracy: 50.1 -> 50.7 | No clear effect |
| Add STGC to MoE-LLaVA-4Top1 (ablation) | MoE-LLaVA, StableLM-1.6B · Avg | score: 53.5 -> 54.6 | Helped |
| Increase training data from 665K to 1021K and add STGC (ablation) | MoE-LLaVA, Phi-2-2.7B, 4Top2, siglip · Avg | score: 63.8 -> 64.9 | Helped |
| Replace token-gradient routing with embedding-based sample routing (ablation) | MoE-LLaVA, StableLM-1.6B, 4Top2 · Avg | score: 57.3 -> 56.9 | Hurt |
| Replace token-gradient routing with feature-based sample routing (ablation) | MoE-LLaVA, StableLM-1.6B, 4Top2 · Avg | score: 57.3 -> 57.2 | Hurt |
| Replace token-gradient routing with task-based sample routing (ablation) | MoE-LLaVA, StableLM-1.6B, 4Top2 · Avg | score: 57.3 -> 48.9 | Hurt |
| Set conflict threshold tau to 0.1 rather than 0.0 (ablation) | STGC on MoE-LLaVA, StableLM-1.6B, 4Top2 · GQA | accuracy: 60.9 -> 60.6 | No clear effect |
| Set conflict threshold tau to -0.1 rather than 0.0 (ablation) | STGC on MoE-LLaVA, StableLM-1.6B, 4Top2 · GQA | accuracy: 60.9 -> 60.6 | No clear effect |
| Increase load-balance loss weight alpha from 0.01 to 0.1 (ablation) | MoE-LLaVA, StableLM-1.6B, 4Top2 · Avg | score: 57.3 -> 57.2 | Hurt |
Surprising findings
- Sample-level routing strategies (embedding-based, feature-based, task-based) often fail to improve or degrade MoE-LLaVA performance, while token-level gradient routing improves it.
- Increasing the load balance loss weight alpha from 0.01 to 0.1 slightly degrades average performance (57.3 to 57.2).
Limitations
- A token's gradient only reflects conflict with its current expert, not the optimal alternative expert, so the optimal routing is not determined.
- Gradient identification uses only bias gradients as an approximation to full weight gradients for memory reasons.
- Gains are modest (around 0.6 to 1.1 average points) and some benchmarks show decreases.
Experimental setup
Datasets: LLaVA-mix-665k Open-LLaVA-NeXT 1021K ScienceQA VQAv2 GQA VizWiz TextVQA POPE MME MMBench MM-Vet AI2D
Benchmarks: VQAv2 GQA VizWiz ScienceQA-IMG TextVQA POPE MME MMBench MM-Vet AI2D ChartQA DocVQA
Metrics: accuracy score average across benchmarks
Baselines: MoE-LLaVA DYNMOE-LLaVA LLaVA-1.5 MoCLE-style embedding-based routing LoRA-MoE-style feature-based routing MoLA-style task-based routing
Builds on: MoE-LLaVA DYNMOE LoRA-MoE MoCLE PCGrad
Closest papers by meaning
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.91)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.90)
- Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (NeurIPS 2024 · similarity 0.89)
- Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · similarity 0.89)
- Mixture of Tokens: Continuous MoE through Cross-Example Aggregation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.