Research map / Vision-language and multimodal / Multimodal LLMs

VL-RouterBench: A Benchmark for Vision-Language Model Routing

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VL-RouterBench, a log-derived benchmark of 14 VQA/STEM/OCR datasets with 17 VLMs (15 open, 2 API) for evaluating vision-language model routers under accuracy-cost trade-offs, and benchmarks 10 routing methods with a soft-label cost-aware training objective.

Problem

Existing multi-model routing benchmarks focus on text LLMs; there is no systematic, reproducible benchmark for routing among vision-language models that accounts for accuracy, cost, and throughput across diverse visual tasks.

Approach

Build quality (correctness) and cost matrices over sample-model pairs from VLMEvalKit inference logs, train routers with an accuracy-cost-aware soft-label cross-entropy (exponential cost decay over correct models), and summarize performance with a harmonic-mean Rank Score of normalized cost and accuracy.

Key results

• RouterDC attains the best Rank Score among routers and reaches accuracy close to the strongest single model at about one fifth of its cost.
• Normalize-Concat fusion gives the best Rank Score (74.05) among fusion methods on MLP with lambda=100.
• A substantial gap remains between the best learned routers and the Oracle, which selects the cheapest correct model per sample.

Surprising findings

• Only-Image routing and Only-Text routing achieve competitive Rank Scores compared with multimodal fusion variants, while GMU attains highest accuracy but very high cost.

Limitations

• Only single-image inputs with rule-based correctness are considered.
• Binary correct/incorrect outcome labels; no continuous quality scores.
• Clean standardized logs; no robustness testing under noisy or perturbed inputs.

Experimental setup

Datasets: MMBench MMStar MMMU RealWorldQA InfoVQA HallusionBench MathVista MathVision MathVerse AI2D ChartQA DocVQA

Benchmarks: VL-RouterBench

Metrics: Avg. Accuracy Avg. Cost Throughput Rank Score (harmonic mean of normalized cost and accuracy) Pareto frontier

Baselines: Oracle Strongest Cheapest KNN PRkNN OVR K-means Linear MLP CosineCls RouterDC Zooter

Builds on: RouterBench RouterEval RouterDC Zooter VLMEvalKit

Closest papers by meaning

• VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models (CVPR 2025 · similarity 0.87)
• What matters when building vision-language models? (NeurIPS 2024 · similarity 0.87)
• UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.87)
• VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.87)
• Olympus: A Universal Task Router for Computer Vision Tasks (CVPR 2025 · similarity 0.86)
• Evaluating Vision-Language Models as Evaluators in Path Planning (CVPR 2025 · similarity 0.86)