Research map / Vision-language and multimodal / Multimodal LLMs
Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Meteor is an efficient LLVM that embeds long multifaceted rationales with a 130M Mamba encoder via a "traversal of rationale" scheme of evenly spaced <tor> tokens, then feeds the embeddings into a 7B InternLM2-based multimodal LM to answer questions without rationale at inference.
Problem
Open-source LLVMs rely on scaling model size, extra vision encoders, or external CV models to close the gap with closed-source models, while the multifaceted rationale that underlies diverse capabilities is not exploited efficiently; long rationales are also hard to embed and are unavailable at infe
Approach
A Mamba model embeds long API-generated rationales, which are inserted via evenly distributed <tor> tokens and trained autoregressively to reproduce rationale segments; its <tor> output features are passed through a tor projector into the backbone MLM, which answers without rationale text at inference.
Key results
- Meteor-7B reaches 2229 on MME, 82.9 on MMBench, and 57.3 on MM-Vet, outperforming listed open-source LLVMs in Table 2.
- Meteor-7B scores 52.8 average on MMStar, above LLaVA-NeXT-34B (52.1), and 21.7 on MathVerse versus 16.8 for SPHINX-MoE-7B×8.
- Ablation: evenly distributed 10 <tor> tokens give MMB 82.9 / MM-Vet 57.3, versus 76.1 / 47.9 with 2 tokens and 74.3 / 45.4 for random placement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add rationale embedding with Meteor-Mamba (ablation) | Meteor-7B · MMB | score: 73.2 -> 82.9 | Helped |
| add rationale embedding with Meteor-Mamba (ablation) | Meteor-7B · MM-Vet | score: 44.8 -> 57.3 | Helped |
| use rationale without Meteor-Mamba (ablation) | Meteor-7B · MMB | score: 82.9 -> 77.0 | Hurt |
| use Mamba to embed answers instead of rationale (ablation) | Meteor-7B · MMB | score: 82.9 -> 74.0 | Hurt |
| use 10 <tor> tokens rather than 5 (ablation) | Meteor-7B · MMB | score: 82.2 -> 82.9 | Helped |
| use 10 <tor> tokens rather than 5 (ablation) | Meteor-7B · MM-Vet | score: 55.8 -> 57.3 | Helped |
| distribute <tor> tokens evenly instead of placing them at the start (ablation) | Meteor-7B · MM-Vet | score: 48.2 -> 57.3 | Helped |
| increase question-rationale training data from 0% to 90% (ablation) | Meteor-7B · MMB | score: 74.0 -> 82.9 | Helped |
| increase Meteor-Mamba size from 130M to 1.4B (ablation) | Meteor · MM-Vet | score: 57.3 -> 58.8 | Helped |
| add Meteor-Mamba (ablation) | Meteor-7B · LLaVA-Bench in the Wild | score: 73.7 -> 87.1 | Helped |
| use Meteor rather than the strongest listed baseline | vision-language models · POPE | score: 88.8 -> 88.7 | No clear effect |
| use Meteor rather than the strongest listed baseline | vision-language models · MMStar | Avg: 52.1 -> 52.8 | Helped |
Surprising findings
- Removing the rationale-trained Mamba and training the MLM directly on rationales (without Mamba) lowered performance below the full Meteor, but a Mamba trained to embed answers instead of rationales also underperformed, indicating the gain comes from rationale embedding.
- Meteor-Mamba output features for questions with and without rationale show high cosine similarity on the diagonal, suggesting embeddings encode rationale information even without explicit rationale text.
Limitations
- Requires multiple GPUs (48GB and 32GB VRAM) for training and inference without (Q)LoRA or quantization, so it is not efficient for users without high-end GPUs.
- Rationales depend on Claude Haiku generation and GPT-4V scoring plus human review, making the pipeline costly and API-dependent.
- No error bars or statistical significance reported; training used only one epoch per step.
Experimental setup
Datasets: ShareGPT4V-Instruct ShareGPT4V-Caption MiniGemini-Instruct Doc-Downstream/Reason GLLaVA-Align/Instruct MathVision MathInstruct/MathPlus LLaVA-Instruct-665K
Benchmarks: Q-Bench SQA-IMG AI2D ChartQA SEED-IMG POPE HallusionBench MME MathVista MMBench MMBench-CN MM-Vet
Metrics: accuracy score
Baselines: LLaVA-NeXT-13B SPHINX-MoE-7B×8 MM1-MoE-7B×32 MoAI CoLLaVO ShareGPT4V-7B InternLM-XC-7B Qwen-VL-Chat-7B
Builds on: Mamba LLaVA CLIP QLoRA Chain-of-Thought
Closest papers by meaning
- Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiency (CVPR 2025 · similarity 0.87)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.87)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.87)
- MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine (ICLR 2025 · similarity 0.86)
- Thinking with Programming Vision: Towards a Unified View for Thinking with Images (CVPR 2026 · similarity 0.86)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.