Research map / Vision-language and multimodal / Multimodal LLMs
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically explores mixture-of-vision-encoders design for MLLMs and finds simple channel concatenation plus a per-expert Pre-Alignment stage yields the Eagle family that beats open-source baselines.
Problem
Mixture-of-vision-encoder MLLMs have been proposed with many fusion strategies, but there is no systematic apples-to-apples comparison of expert selection, fusion method, and training recipe, and vision-only encoders integrate poorly with LLMs.
Approach
Shows that simple channel concatenation of complementary vision experts (CLIP, ConvNeXt, SAM, Pix2Struct, EVA-02) matches complex fusion methods, unfreezes encoders during training, and adds a Pre-Alignment stage that aligns each expert individually with a frozen LLM before joint training.
Key results
- Eagle-X5 (+Pre-Align) with Vicuna-13B reaches 1605 MME, 71.6 MMBench, 42.7 MathVista, 89.2 POPE, and 598 OCRBench.
- Channel concatenation of CLIP+ConvNeXt matches sequence append (681.5 vs 690.5 avg in the 2-encoder setting) while using 1024 instead of 2048 visual tokens.
- Using the same Cambrian-1 data with Llama3-8B, Eagle-X5 averages 65.2 on Knowledge, 74.0 on General, and 77.0 on OCR/Chart vs Cambrian-1's 61.3, 73.1, and 71.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Unfreeze the interpolated CLIP encoder during SFT instead of keeping it frozen (ablation) | CLIP-448 MLLM · CLIP 448 interpolation | Avg.: 589.7 -> 670.5 | Helped |
| Increase interpolated CLIP input resolution from 448 to 672 (ablation) | CLIP MLLM with unlocked encoder · CLIP high-resolution adaptation | Avg.: 670.5 -> 674.2 | No clear effect |
| Use channel concatenation rather than Mini-Gemini fusion (ablation) | CLIP + ConvNeXt · OCRBench | score: 478 -> 551 | Helped |
| Replace channel concatenation with sequence append fusion (ablation) | CLIP + ConvNeXt · Avg. | average score: 681.5 -> 690.5 | Helped |
| Add EVA-02 to CLIP + ConvNeXt (ablation) | Vision expert selection · POPE | score: 88.3 -> 89.0 | Helped |
| Add DINOv2 to CLIP + ConvNeXt + EVA-02 + Pix2Struct + SAM (ablation) | Vision expert selection · Avg. | average score: 697.1 -> 686.8 | Hurt |
| Add pre-alignment of individual experts (ablation) | Eagle-X5 · MME perception | score: 1528 -> 1582 | Mixed |
| Unlock vision encoders during SFT (ablation) | Vision-expert MLLMs · Avg. | average score: 635.0 -> 659.7 | Helped |
| Train for 2 SFT epochs instead of 1 (ablation) | Eagle · Avg. | average score: 697.1 -> 698.3 | Helped |
| Unlock vision encoders during pretraining (ablation) | Eagle without pre-alignment · Avg. | average score: 697.1 -> 698.0 | Helped |
| Add Eagle1.8M to the pretraining data (ablation) | Eagle without pre-alignment · Avg. | average score: 698.0 -> 699.5 | Helped |
| Pre-align experts using LLaVA-595k + Eagle1.8M rather than Eagle1.8M alone (ablation) | Eagle · Avg. | average score: 707.1 -> 707.8 | Helped |
Surprising findings
- Recently proposed injection-based fusion methods (LLaVA-HR, Mini-Gemini, deformable attention) often underperform ConvNeXt alone on TextVQA and OCRBench.
- Freezing vision encoders, a common LLaVA-family choice, is suboptimal; unlocking them during training gives consistent gains, especially with multiple experts.
Limitations
- Greedy round-robin expert selection explores only a limited subset of combinations.
- Detailed hardware and compute budget are not stated.
- Eagle1.8M data composition is not released as a dataset in the text.
Experimental setup
Datasets: LLaVA-595k Eagle1.8M DocVQA synDog-EN ChartQA DVQA AI2D ShareGPT-4V LVIS-Instruct4V LRV-Instruct Geo170k LLaVAR
Benchmarks: GQA VQAv2 VizWiz MME MMBench SEED MathVista MMMU POPE ScienceQA OCRBench TextVQA
Metrics: accuracy F1 perception score normalized average
Baselines: LLaVA-1.5 LLaVA-NeXT InternVL LLaVA-HR Monkey Mini-Gemini LLaVA-UHD Cambrian-1 MGM-HD
Builds on: LLaVA-1.5 Prismatic VLM Mini-Gemini LLaVA-HR Cambrian-1
Closest papers by meaning
- Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs (NeurIPS 2026 · similarity 0.91)
- Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders (ICLR 2026 · similarity 0.90)
- VCoder: Versatile Vision Encoders for Multimodal Large Language Models (CVPR 2024 · similarity 0.90)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.90)
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.89)
- DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.