Research map / Vision-language and multimodal / Human and embodied modeling
R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R2-T2, a training-free test-time method that re-optimizes an MoE multimodal model's routing weights for each test sample using reference samples' neighborhoods (neighborhood gradient descent, kernel regression, mode finding).
Problem
End-to-end trained routers in multimodal mixture-of-experts LMMs do not always give optimal expert weights for individual test samples, especially on unseen or out-of-distribution tasks, limiting performance.
Approach
Adjust each test sample's routing-weight vector at inference by moving it toward routing weights of correctly predicted nearby reference samples (in a task-embedding space), using a neighborhood loss surrogate with gradient descent or gradient-free kernel regression/mean-shift, without updating any model parameters.
Key results
- R2-T2 (kNN, NGD) lifts MoAI-7B on MMBench from 79.3 to 85.2 and MME-P from 1714 to 1785.5.
- R2-T2 (MoVA-7B) improves TextVQA from 76.4 to 83.2 and GQA from 64.8 to 71.5, outperforming listed 8/13/34B VLMs on most benchmarks.
- Oracle routing reaches 92.1 on MMBench for MoAI, so the method captures roughly 70-80% of the possible gain without labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply test-time NGD re-routing | MoAI-7B · MMBench | Accuracy (%): 79.3 -> 85.2 | Helped |
| Apply test-time NGD re-routing | MoAI-7B · MME-P | Accuracy score: 1714.0 -> 1785.5 | Helped |
| Apply test-time NGD re-routing | MoAI-7B · TextVQA | Accuracy (%): 67.8 -> 73.5 | Helped |
| Apply test-time NGD re-routing | MoVA-7B · MMBench | Accuracy (%): 74.3 -> 81.2 | Helped |
| Use mode finding instead of the base router | MoAI-7B · MMBench | Accuracy (%): 79.3 -> 80.8 | Helped |
| Use NGD instead of mode finding | MoAI-7B · MMBench | Accuracy (%): 80.8 -> 85.2 | Helped |
| Use kNN with k=5 rather than k=20 (ablation) | R2-T2 NGD on MoAI · MoAI average across 8 benchmarks | Average score: 76.6 -> 80.7 | Helped |
| Replace the Gaussian kernel with a linear kernel (ablation) | R2-T2 NGD on MoAI · MoAI average across 8 benchmarks | Average score: 80.7 -> 76.3 | Hurt |
| Use NV-Embed-V2 rather than Sentence-Bert (ablation) | R2-T2 NGD on MoAI · MoAI average across 8 benchmarks | Average score: 77.5 -> 80.7 | Helped |
| Increase NGD steps from 5 to 10 (ablation) | R2-T2 NGD on MoAI · MoAI average across 8 benchmarks | Average score: 76.6 → 80.7 | Helped |
| Increase NGD steps from 10 to 20 (ablation) | R2-T2 NGD on MoAI · MoAI average across 8 benchmarks | Average score: 80.7 -> 80.5 | No clear effect |
| Replace cosine learning-rate schedule with fixed (1e-3) (ablation) | R2-T2 NGD on MoAI · MMBench | Accuracy (%): 85.2 -> 71.8 | Hurt |
Surprising findings
- Oracle routing improves accuracy by 10% or more on most benchmarks, showing that the end-to-end router is far from optimal per sample.
Limitations
- Test-time cost rises by 6-7x FLOPs for the best gradient-based variant.
- Gains depend on the quality and coverage of the reference set; some correct predictions flip to incorrect (2.31%).
Experimental setup
Datasets: VQA-V2 Visual7W COCO-QA CLEVR A-OKVQA TQA MathVista ST-VQA DocVQA
Benchmarks: MMBench MME-P SQA-IMG AI2D TextVQA GQA CVBench-2D CVBench-3D PhysBench
Metrics: accuracy MME-P score (Accuracy + Accuracy+)
Baselines: MoAI-7B MoVA-7B Mode Finding Kernel Regression Oracle routing Cambrian1-34B LLaVA-NeXT-34B Mini-Gemini-HD-34B
Builds on: MoAI MoVA Mixture-of-Experts
Closest papers by meaning
- MoVA: Adapting Mixture of Vision Experts to Multimodal Context (NeurIPS 2024 · similarity 0.88)
- Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models (ICLR 2026 · similarity 0.88)
- CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts (NeurIPS 2024 · similarity 0.88)
- Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders (ICLR 2026 · similarity 0.87)
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.87)
- MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.