Research map / Vision-language and multimodal / Multimodal LLMs
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents SPHINX-X, a family of MLLMs built on SPHINX with fewer visual encoders (MoV), skip tokens for fully-padded sub-images, one-stage all-in-one training, a large multi-domain dataset with OCR and Set-of-Mark data, and LLM backbones from 1.1B to 8x7B.
Problem
Open-source MLLMs have limited task-domain data coverage (weak on OCR, charts, documents, math) and are mostly built on 7B/13B dense LLMs, leaving data and parameter scaling under-explored; SPHINX's architecture and two-stage training are also costly.
Approach
Streamline SPHINX by keeping only two complementary vision encoders (DINOv2 and CLIP-ConvNeXt, called MoV), replacing fully-padded sub-image tokens with learnable skip tokens, and training all LLM and projection parameters in one stage on a large multi-domain mixture, then pairing this recipe with LLMs of varying scale.
Key results
- SPHINX-MoE reaches 41.0 vs 15.6 for open-source LLaVA-NeXT on MathVerse-All... (text in paper: SPHINX-MoE 15.6 on MathVerse All, best among open-source models; GPT-4V 41.0).
- SPHINX-MoE achieves 62.7 on VQAv2-style academic VQA column and 61.9 VizWiz, surpassing SPHINX on most of the 6 listed VQA benchmarks.
- SPHINX-Plus scores 90.11 on RefCOCOg val-u and 92.44 on RefCOCO val, the best among SPHINX-X variants on referring expression comprehension.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale up data and move from LLaMA2-13B SPHINX to SPHINX-Plus (one-stage, larger dataset) | SPHINX-Plus · Popular MLLM benchmarks | MM-Vet: 36.6 -> 47.9 | Helped |
| Scale up data with one-stage training (SPHINX-Plus vs SPHINX) | SPHINX-Plus · Popular MLLM benchmarks | MMBench: 67.1 -> 71.0 | Helped |
| Scale up base LLM from TinyLlama-1.1B to InternLM2-7B to Mixtral-8x7B (MoE) under same pipeline | SPHINX-X family · Popular MLLM benchmarks | MMBench: 56.6 (Tiny) -> 57.9 (Intern2) -> 71.3 (MoE) | Helped |
| Use Mixtral-8x7B MoE LLM (SPHINX-MoE) vs LLaMA2-13B SPHINX-Plus | SPHINX-MoE · MathVerse | All accuracy: SPHINX-MoE 15.6 vs GPT-4V 41.0 (open-source best LLaVA-NeXT 15.6) | Mixed |
| Train on image-only data and feed middle video frame at inference | SPHINX-Plus · Video-Bench | Avg.: 45.1 vs best video-trained Video-ChatGPT 38.3 | Helped |
| Image-only training with middle-frame video input on MVBench | SPHINX-Plus · MVBench | Avg.: 39.7 vs VideoChat2 51.1 | Hurt |
| Increase input resolution to 672x672 with 3x3 sub-images (SPHINX-Plus-2K) | SPHINX-Plus · Text-oriented VQA (TextVQA) | TextVQA accuracy: 65.7 -> 70.6 | Helped |
| Scale data and one-stage training from SPHINX to SPHINX-Plus | SPHINX-Plus · REC RefCOCO val | Top-1 Acc@0.5: 91.05 -> 92.44 | Helped |
| Use Tiny LLM (TinyLlama-1.1B) instead of 13B LLaMA2 (ablation) | SPHINX-Tiny · Popular MLLM benchmarks | SEED: 71.6 (SPHINX) -> 17.1 | Hurt |
| Activate 2 vs other numbers of experts at inference in SPHINX-MoE (ablation) | SPHINX-MoE · MME and other benchmarks | performance | Mixed |
| Random pruning of experts in SPHINX-MoE layers (ablation) | SPHINX-MoE · Expert pruning analysis | performance | Hurt |
| Remove CLIP-ViT and Q-former encoders (keep DINOv2 and CLIP-ConvNeXt) | SPHINX-X · Efficiency design choice | performance | No clear effect |
Surprising findings
- Experts in SPHINX-MoE show no clear domain or modality specialization, and pruning specific experts can cause large performance loss.
- SPHINX-Plus, an image-only model with no video training, outperforms video-specific MLLMs on Video-Bench average (45.1 vs 38.3 for Video-ChatGPT).
Limitations
- Weak performance on multi-discipline tasks such as MMMU and CMMMU, attributed to lack of multi-disciplinary multimodal training data.
- Video results rely on single middle-frame sampling without video training, underperforming on temporal tasks such as MOT.
- SPHINX-Tiny scores poorly on some benchmarks (e.g., SEED 17.1).
Experimental setup
Datasets: UltraChat OpenOrca MetaMathQA MathInstruct WizardCoder COCO Objects365 OpenImages LVIS VQAv2 GQA TextVQA
Benchmarks: POPE MME MMBench SEED-Bench LLaVA-Bench MM-Vet MathVista VQAv2 GQA TextVQA DocVQA ChartQA
Metrics: accuracy score
Baselines: SPHINX LLaVA1.5-13B Qwen-VL-7B-Chat GPT-4V Gemini-Pro MiniGPT-v2
Builds on: SPHINX LLaMA2 Mixtral-8x7B TinyLlama InternLM2
Closest papers by meaning
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.91)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.90)
- HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.90)
- LRM-LLaVA: Overcoming the Modality Gap of Multilingual Large Language-Vision Model for Low-Resource Languages (AAAI 2025 · similarity 0.90)
- LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.89)
- DiM$^3$: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.