Research map / Vision-language and multimodal / Multimodal LLMs
Octopus: A Multi-modal LLM with Parallel Recognition and Sequential Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Octopus, an MLLM that runs object-query-based DETR recognition inside the bottom LLM layers in parallel, then passes recognition results to top layers for sequential understanding, using index tokens instead of textual box coordinates.
Problem
Current MLLMs generate recognition outputs (e.g., bounding box coordinates, polygons) token-by-token in the same sequential LLM head as understanding, which is slow and mismatched to numerical regression-style recognition.
Approach
Split the LLM head into bottom layers that perform parallel recognition via object queries and a light DETR decoder, and top layers that perform sequential understanding by emitting index tokens that select pre-detected boxes, trained end-to-end with a DETR Hungarian loss plus language modeling loss and dynamic index-token matching.
Key results
- Octopus (224 res) achieves 5x faster grounding inference than Shikra (0.17s vs 0.89s per RefCOCO record; 0.82s vs 3.80s on Flickr30k) and higher REC accuracy, e.g. +1.51% on RefCOCO+ test-A.
- Octopus-7B at 336 resolution reaches 86.19 on RefCOCOg test and 79.2 VQAv2, 63.3 GQA, 50.1 VizWiz.
- Octopus at 224 resolution reaches 66.2 on MMBench and 84.8 on POPE, beating Shikra (58.8, 83.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace sequential recognition with parallel recognition using object queries and a DETR decoder | Octopus vs. Shikra · RefCOCO+ test-A | accuracy: 87.36 -> 89.40 | Helped |
| replace sequential recognition with parallel recognition using object queries and a DETR decoder | Octopus vs. Shikra · RefCOCO | inference time: 0.89 -> 0.17 | Helped |
| replace sequential recognition with parallel recognition using object queries and a DETR decoder | Octopus vs. Shikra · Flickr30k | inference time: 3.80 -> 0.82 | Helped |
| place DETR after 12 rather than 16 LLM layers (ablation) | Octopus · RefCOCOg val | accuracy: 79.5 -> 78.7 | Hurt |
| place DETR after 14 rather than 16 LLM layers (ablation) | Octopus · RefCOCOg val | accuracy: 79.5 -> 79.4 | Hurt |
| place DETR after 18 rather than 16 LLM layers (ablation) | Octopus · RefCOCOg val | accuracy: 79.5 -> 79.1 | Hurt |
| place DETR after 20 rather than 16 LLM layers (ablation) | Octopus · RefCOCOg val | accuracy: 79.5 -> 78.2 | Hurt |
| use 48 object queries rather than 64 (ablation) | Octopus · RefCOCOg val | accuracy: 79.5 -> 78.1 | Hurt |
| use 80 object queries rather than 64 (ablation) | Octopus · RefCOCOg val | accuracy: 79.5 -> 79.5 | No clear effect |
| use 100 object queries rather than 64 (ablation) | Octopus · RefCOCOg val | accuracy: 79.5 -> 77.9 | Hurt |
| use 120 object queries rather than 64 (ablation) | Octopus · RefCOCOg val | accuracy: 79.5 -> 76.5 | Hurt |
| relay intermediate DETR recognition results to the top LLM layers, which select a final box | Octopus · RefCOCOg | accuracy: +1.18% and + 2.12% | Helped |
Surprising findings
- Parallel recognition in the bottom layers improves accuracy on non-grounding tasks such as GQA, VizWiz and SciQA even though those tasks do not ask for box outputs.
- The LLM top layers improve recognition accuracy over the rank-1 DETR output (e.g. 83.11 vs 81.93 at 224 resolution).
Limitations
- Not explored scaling to larger LLMs or more open-ended instruction-tuning data due to limited GPU resources.
- Referring segmentation results are preliminary, using only 224 resolution (63.6 cIoU on RefCOCO val) and well below LISA (74.6).
Experimental setup
Datasets: LLaVA pretraining data RefCOCO RefCOCO+ RefCOCOg Flickr30k LLaVA-Instruct Visual Genome
Benchmarks: RefCOCO RefCOCO+ RefCOCOg VQAv2 OKVQA GQA VizWiz SciQA MMBench LLaVA-Bench SEED MM-Vet
Metrics: accuracy cIoU inference time FPS rank-k recall
Baselines: Shikra MiniGPT-v2 OFA-L VisionLLM LLaVA-1.5 Qwen-VL-Chat InstructBLIP G-DINO-L UNINEXT-H LISA
Builds on: DETR LLaVA Shikra
Closest papers by meaning
- Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models (ICLR 2025 · similarity 0.87)
- Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.86)
- MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.86)
- MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs (ICML 2026 · similarity 0.86)
- VCoder: Versatile Vision Encoders for Multimodal Large Language Models (CVPR 2024 · similarity 0.86)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.