Research map / Vision-language and multimodal / Multimodal LLMs
Honeybee: Locality-enhanced Projector for Multimodal LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Honeybee, an MLLM with locality-enhanced visual projectors (C-Abstractor and D-Abstractor) that are flexible in visual token count, plus a systematic study of instruction-data mixing, balancing, and instructization templates.
Problem
Visual projectors in MLLMs are underexplored; linear projectors preserve local context but are inflexible in token count, while abstractors (resampler, Q-former) are flexible but lose local/spatial detail, and multifaceted instruction data is poorly utilized.
Approach
Injects locality into abstractors: C-Abstractor uses ResNet blocks before and after adaptive average pooling; D-Abstractor uses deformable attention with uniformly initialized reference points and pooled query initialization, keeping flexible token counts while preserving local context.
Key results
- Honeybee 7B (M=144) reaches MMB 70.1, MME-perception 1584.2, SEED-Image 64.5 and LLaVA-Bench 67.1, versus LLaVA-1.5 7B at 64.3 MMB and 1510.7 MME-perception.
- Honeybee 13B (M=576) reaches MMB 73.6, MME-perception 1661.1, SEED-Image 68.6 and LLaVA-Bench 77.5, exceeding previous SoTA reported in the paper (e.g., LLaVA-1.5 13B MMB 67.7).
- At equal M=256, C-Abstractor outperforms the linear projector on six spatial understanding tasks (Avg^N 56.3 vs 52.6) and runs faster per step than linear at M=144 versus 256 (2.23 vs 3.04 s/step).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use C-Abstractor instead of Resampler at M=144 | Honeybee projector · Spatial understanding tasks (six-task Avg^N) | normalized average: 43.9 -> 53.5 | Helped |
| Use C-Abstractor instead of Linear at M=256 | Honeybee projector · Spatial understanding tasks (six-task Avg^N) | normalized average: 52.6 -> 56.3 | Helped |
| Increase visual tokens from M=144 to M=256 (ablation) | Honeybee C-Abstractor (7B) · SEED-Bench Image-only | accuracy: 64.5 -> 65.5 | Helped |
| Increase visual tokens from M=256 to M=576 (ablation) | Honeybee C-Abstractor (13B) · LLaVA-Bench (In-the-Wild) | score: 75.7 -> 77.5 | Helped |
| Use one dataset per task type instead of diverse datasets per task (ablation) | Honeybee instruction tuning · MMBench-dev | accuracy: 69.2 -> 67.4 | Hurt |
| Remove open-ended VQA data (ablation) | Honeybee instruction tuning · MME perception | score: 1568 -> 1310 | Hurt |
| Remove multiple-choice VQA data (ablation) | Honeybee instruction tuning · MMBench-dev | accuracy: 69.2 -> 30.4 | Hurt |
| Remove captioning data (ablation) | Honeybee instruction tuning · LLaVA-Bench (In-the-Wild) | score: 64.5 -> 59.8 | Hurt |
| Remove GPT-assisted visual instruction data (ablation) | Honeybee instruction tuning · LLaVA-Bench (In-the-Wild) | score: 64.5 -> 51.9 | Hurt |
| Remove existing vision-language task data (ablation) | Honeybee instruction tuning · MME perception | score: 1568 -> 1123 | Hurt |
| Replace fine-grained dataset-specific templates with coarse-grained task templates (ablation) | Honeybee instructization · MMBench-dev | accuracy: 69.2 -> 68.9 | Hurt |
| Use multiple templates instead of a single template (ablation) | Honeybee instructization · LLaVA-Bench (In-the-Wild) | score: 64.5 -> 61.0 | Hurt |
Surprising findings
- Adding an image indicator token hurts performance (Avg^N 70.6 without vs 69.0 with), and LoRA tuning of the LLM is far worse than full fine-tuning.
- Increasing template diversity (multiple templates per dataset, with or without input inversion) does not improve performance over a single fine-grained template per dataset.
Limitations
- Evaluation focuses on image-based benchmarks with a single vision encoder (CLIP ViT-L/14); the projector is not tested with other encoders or modalities.
- Relies on GPT-4 judging for LLaVA-Bench and on hand-tuned dataset balancing that requires ablation runs.
- Some ablations use a short training schedule and a single LLM size, so conclusions may not transfer to all settings.
Experimental setup
Datasets: BlipCapFilt COYO100M VQAv2 GQA OCRVQA VSR ScienceQA A-OKVQA RefCOCO RefCOCO+ RefCOCOg Visual Genome
Benchmarks: MME MMBench SEED-Bench LLaVA-Bench (In-the-Wild) ScienceQA MM-Vet MMMU POPE
Metrics: accuracy MME perception score normalized average Avg^N GPT-4 judged score seconds per pre-training step
Baselines: LLaVA-1.5 Qwen-VL-Chat InstructBLIP BLIP-2 MiniGPT-4 mPLUG-Owl LLaMA-AdapterV2 Shikra
Builds on: LLaVA Vicuna CLIP Deformable DETR Qwen-VL resampler
Closest papers by meaning
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.88)
- Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models (CVPR 2026 · similarity 0.87)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.86)
- Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation (NeurIPS 2025 · similarity 0.86)
- Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts (AAAI 2025 · similarity 0.86)
- Dense Connector for MLLMs (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.