Research map / Vision-language and multimodal / Multimodal LLMs
Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Cobra replaces the Transformer LLM in a LLaVA-style MLLM with pretrained Mamba language models (2.8B and 7B), fusing DINOv2 and SigLIP features, to get linear-complexity, faster multimodal inference.
Problem
Transformer-based MLLMs have quadratic computational complexity, making high-speed inference on edge devices costly; small-LLM MLLMs trade away accuracy for speed.
Approach
Use a pretrained Mamba (selective SSM) language model as the backbone of a LLaVA-style MLLM, with a fused DINOv2+SigLIP vision encoder and an MLP or LDPv2 projector, fully fine-tuned on a combined ~1.2M-sample instruction dataset, to obtain fast linear-time inference with competitive accuracy.
Key results
- Cobra-3.5B (Mamba-2.8B) reaches 166.47 tokens/s versus 49.50 for MobileVLM v2 and 20.33 for MoE-LLaVA on a single A100, with 3.5B total parameters.
- Cobra-8B (Mamba-7B) scores 79.2 VQA-v2, 63.9 GQA, 56.2 VizWiz, 59.5 TextVQA, 62.9 VSR, 87.6 POPE, exceeding LLaVA v1.5-7B on all listed benchmarks.
- Cobra-3.5B improves VSR by 6.9 points and POPE by 2.5 points over LLaVA v1.5-7B while using about 48% of its parameters, though it trails on RefCOCO-family grounding (avg 48.4 vs 51.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use fused DINOv2 and SigLIP features instead of SigLIP-only features (ablation) | Cobra-3.5B · VSR | accuracy: 53.2 -> 58.4 | Helped |
| replace the projector with LDPv2 (ablation) | Cobra-3.5B · TextVQA | accuracy: 58.2 -> 54.7 | Hurt |
| use the chat-tuned Mamba model instead of the base model (ablation) | Cobra-3.5B · VizWiz | accuracy: 47.2 -> 49.7 | Helped |
| use the chat-tuned Mamba model instead of the base model (ablation) | Cobra-3.5B · GQA | accuracy: 62.7 -> 62.3 | Hurt |
| fine-tune for two epochs instead of one (ablation) | Cobra-3.5B · VSR | accuracy: 53.8 -> 58.4 | Helped |
| add pre-alignment before fine-tuning rather than use a randomly initialized projector (ablation) | Cobra-3.5B · RefCOCO+ | accuracy: 45.6 -> 29.7 | Hurt |
| place OCR tokens before the question instead of after it | Cobra-3.5B · TextVQA | accuracy: 43.0 -> 58.2 | Helped |
| replace the projector with LDPv2 (ablation) | Cobra-3.5B · image description inference | tokens/s: 166.47 -> 166.85 | No clear effect |
| use Cobra-3.5B instead of LLaVA v1.5 | MLLM · VSR | accuracy: 51.5 -> 58.4 | Helped |
| use Cobra-8B instead of LLaVA v1.5 | MLLM · RefCOCO | accuracy: 55.1 -> 58.2 | Helped |
Surprising findings
- Placing OCR tokens after the question (LLaVA default) degraded TextVQA performance below the no-OCR setting for Mamba models, which the authors attribute to an RNN inductive bias; placing OCR tokens first fixed this.
- Lightweight down-sampling projector (LDPv2) gave no meaningful speedup in Cobra, since prompt processing is not the bottleneck for the parallel RNN formulation.
Limitations
- Lags behind MoE-LLaVA on VQA-v2 and GQA for the small-scale setting by about 1-2 points.
- Grounding performance of Cobra-3.5B is 3-4 points below LLaVA v1.5-7B on RefCOCO-family benchmarks.
- Speed advantage measured only on one GPU with a single prompt and fixed 256 output tokens.
Experimental setup
Datasets: LLaVA v1.5 mixed instruction data LLaVA-Instruct ShareGPT LVIS-Instruct-4V LRV-Instruct VQA-v2 GQA VizWiz TextVQA VSR POPE RefCOCO
Benchmarks: VQA-v2 GQA VizWiz TextVQA VSR POPE RefCOCO RefCOCO+ RefCOCOg
Metrics: accuracy tokens per second total latency (s)
Baselines: LLaVA v1.5 MoE-LLaVA LLaVA-Phi MobileVLM v2 Qwen-VL InstructBLIP BLIP-2 MiniGPT-4
Builds on: Mamba LLaVA v1.5 LLaVA-Phi MobileVLM DINOv2 SigLIP
Closest papers by meaning
- The Mamba in the Llama: Distilling and Accelerating Hybrid Models (NeurIPS 2024 · similarity 0.88)
- LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement (ICLR 2025 · similarity 0.88)
- Block-Biased Mamba for Long-Range Sequence Processing (NeurIPS 2025 · similarity 0.87)
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (ICML 2024 · similarity 0.87)
- Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (NeurIPS 2024 · similarity 0.87)
- Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.