Research map / Multi-agent systems and games / LLM agents
Towards Robust Multi-Modal Reasoning via Model Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M3, a learned model selector for multi-modal multi-step reasoning agents that models input, candidate-model choices, and subtask dependencies on a computation graph, and introduces the MS-GQA benchmark built from VisProg on GQA.
Problem
Model selection for multi-modal agents that chain multiple AI models across dependent subtasks; existing agents use fixed or metric-based model choices, and traditional model selection methods ignore subtask dependencies, making execution fragile.
Approach
Represent the multi-step reasoning as a directed acyclic computation graph whose nodes carry model embeddings and a multi-modal input embedding; a GAT-based computation graph learner predicts execution success for each model-choice configuration, trained with a list-wise categorical cross-entropy loss, and selects the configuration with highest predicted success.
Key results
- On the full MS-GQA test set, the proposed M3 framework reaches 68.70% SER, a 2.69-point improvement over the best prior baseline MetaGL (66.01%).
- A random selector reaches 56.51% SER and the best training-free baseline GlobalBest reaches 63.58% SER.
- Model selection adds about 0.09s overhead versus 0.62s total task execution time, which the authors describe as negligible.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use M3 for input- and dependency-aware model selection instead of MetaGL | model selection on MS-GQA · MS-GQA Full test | SER (%): 66.01_{\pm 0.4} -> 68.70_{\pm 0.6}; +2.69% | Helped |
| use M3 instead of the best baseline | model selection on MS-GQA · MS-GQA Compare | SER (%): 82.26_{\pm 0.0} -> 76.45_{\pm 1.4} | Hurt |
| use M3 instead of the best baseline | model selection on MS-GQA · MS-GQA Verify | SER (%): 75.46_{\pm 0.9} -> 75.09_{\pm 0.6} | Hurt |
| use M3 instead of the best baseline | model selection on MS-GQA · MS-GQA Logical | SER (%): 78.65_{\pm 0.0} -> 77.53_{\pm 2.9} | Hurt |
| add computation graph description text features to NCF | NCF on MS-GQA · MS-GQA Full test | SER (%): 65.92_{\pm 1.5} -> 64.40_{\pm 1.7} | Hurt |
| add computation graph description text features to MetaGL | MetaGL on MS-GQA · MS-GQA Full test | SER (%): 66.01_{\pm 0.4} -> 65.62_{\pm 3.2} | Hurt |
| replace BLiP features with ViLT features (ablation) | M3 · MS-GQA Full test | SER: 68.70_{\pm 0.6} -> 65.91_{\pm 0.7} | Hurt |
| replace BLiP features with BERT+ViT features (ablation) | M3 · MS-GQA Full test | SER: 68.70_{\pm 0.6} -> 64.94_{\pm 0.5} | Hurt |
| replace GAT with GRU (ablation) | M3 · MS-GQA Full test | SER: 68.70_{\pm 0.6} -> 68.02_{\pm 0.7} | Hurt |
| replace GAT with Transformer (ablation) | M3 · MS-GQA Full test | SER: 68.70_{\pm 0.6} -> 65.32_{\pm 0.4} | Hurt |
| replace CCE with BCE loss (ablation) | M3 · MS-GQA Full test | SER: 68.70_{\pm 0.6} -> 67.65_{\pm 1.3} | Hurt |
Surprising findings
- Adding text features from LLM-generated computation graph descriptions (NCF++, MetaGL++) often fails to help and can hurt the original methods.
- In-context LLM model selection (HuggingGPT-style) picked the same model regardless of question structure, so it was excluded from baselines.
Limitations
- Performance declines with missing training data, showing dependence on dataset size.
- Supervision only uses final execution outcomes, not intermediate step results.
- Evaluated only on MS-GQA with the VisProg agent and a small candidate pool (70 configurations).
Experimental setup
Datasets: GQA
Benchmarks: MS-GQA
Metrics: Successful Execution Rate (SER)
Baselines: Random VisProg ExMetric GlobalBest NCF MetaGL NCF++ MetaGL++
Builds on: VisProg NCF MetaGL GAT
Closest papers by meaning
- AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning (ICML 2026 · similarity 0.91)
- AutoTool: Efficient Tool Selection for Large Language Model Agents (AAAI 2026 · similarity 0.89)
- A$^2$FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning (ICLR 2026 · similarity 0.88)
- SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agents (NeurIPS 2025 · similarity 0.88)
- Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation (ICLR 2026 · similarity 0.88)
- Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.