Research map / Vision-language and multimodal / Multimodal LLMs

Olympus: A Universal Task Router for Computer Vision Tasks

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Olympus fine-tunes a Mipha-3B-based MLLM controller on a GPT-4o-generated routing dataset (OlympusInstruct) to emit task-specific routing tokens that delegate 20 vision tasks to specialist models, including chained actions.

Problem

Unified vision systems either rely on all-in-one generative models that are costly to train and conflict across tasks, or on prompt-only LLM controllers like HuggingGPT that do not train for routing; a trained, scalable router over many image, video and 3D tasks is needed.

Approach

Train an MLLM to emit task-specific routing tokens (e.g., <image_edit>...</image_edit>) that invoke external specialist models, handling VQA internally and chaining multiple tool calls within a single instruction, using a large GPT-4o-generated instruction dataset.

Key results

• Single-task routing accuracy of 94.75% on OlympusBench across 20 tasks, versus 81.35% for HuggingGPT (GPT-4o).
• Chain-of-action precision of 91.82% with edit distance 0.18, versus 75.03% precision for HuggingGPT (GPT-4o).
• Human-evaluated success rate of 86.5% on 200 human instructions, versus 75.2% for HuggingGPT (GPT-4o); multimodal benchmark performance is comparable to Mipha-3B (e.g., MMB 71.2 vs 69.7).

Measured effects reported in the paper

Surprising findings

• Adding 0 vs 20 tasks shows a slight degradation in routing accuracy as the number of tasks grows (96.38% at 5 tasks to 94.75% at 20).
• Olympus-3B outperforms its larger 7B and 13B variants on several benchmarks, which the authors attribute to higher resolution and the Mipha visual encoder.

Limitations

• Quality and biases of the GPT-4o-generated training data propagate into the model.
• Routing accuracy degrades slightly as the number of tasks increases.
• Multimodal understanding is only comparable to baselines, with a drop on TextVQA.

Experimental setup

Datasets: OlympusInstruct LLaVA-Mix665K

Benchmarks: VQAv2 GQA VisWiz ScienceQA-IMG TextVQA MME MMBench MM-Vet POPE MMMU OlympusBench

Metrics: accuracy precision recall F1 edit distance success rate

Baselines: HuggingGPT (GPT-4o) HuggingGPT (GPT-4o mini) Mipha-3B LLaVA-1.5 Qwen-VL-Chat mPLUG-Owl2

Builds on: Mipha HuggingGPT LLaVA

Closest papers by meaning

• VL-RouterBench: A Benchmark for Vision-Language Model Routing (CVPR 2026 · similarity 0.86)
• Argus: A Compact and Versatile Foundation Model for Vision (CVPR 2025 · similarity 0.85)
• VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models (CVPR 2025 · similarity 0.84)
• CogVLA: Cognition-Aligned Vision-Language-Action Models via Instruction-Driven Routing & Sparsification (NeurIPS 2025 · similarity 0.84)
• MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.83)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.83)