Research map / Vision-language and multimodal / Multimodal LLMs
"SPHINX: A Mixer of Weights, Visual Embeddings and Image Scales for Multi-modal Large Language Models"
Ziyi Lin, Dongyang Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang and 3 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add high-resolution sub-images (mixed scales, SPHINX-1k vs SPHINX) (ablation) | SPHINX · POPE | accuracy: 80.7 -> 90.8 | Improved |
| add mixed scales with sub-images (ablation) | SPHINX · TextVQA | accuracy: 51.6 -> 58.8 | Improved |
| add sub-images (mixed scales) for referring comprehension (ablation) | SPHINX · RefCOCO+ val | Top-1 accuracy@0.5: 82.77 -> 86.64 | Improved |
| keep low-resolution global view alongside sub-images (ablation) | SPHINX-1k · RefCOCO val | Top-1 accuracy@0.5: 71.72 -> 91.05 | Improved |
| mix all four visual encoders instead of Q-Former global features only (ablation) | SPHINX · RefCOCO val | Top-1 accuracy@0.5: 46.52 -> 89.14 | Improved |
| mix all visual encoders for VQA benchmarks (ablation) | SPHINX · General and text-oriented VQA | accuracy | Mixed |
| jointly train with text-only RefinedWeb during pre-training (ablation) | SPHINX · pre-training loss | text generation loss | Improved |
| mix scales at inference (SPHINX-1k vs SPHINX) cost (ablation) | SPHINX · A100 inference, 200 tokens | time and memory: 5.2s -> 5.5s; 29GB -> 33GB | Hurt |
| go from SPHINX-1k to SPHINX-2k resolution (ablation) | SPHINX · MMBench | accuracy: 67.1 -> 65.9 | Mixed |
Later papers that compared with this method
6 later accepted papers report SPHINX-Tiny as a baseline in their results tables; 6 of them report a better number on most of the results they share with it.
- Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions (ICML 2026 · better on 1 of 1 shared result)
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (ICML 2024 · better on 5 of 6 shared results)
- RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation (NeurIPS 2024 · better on 2 of 2 shared results)
- Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (NeurIPS 2024 · better on 2 of 2 shared results)
- BRAVE: Broadening the visual encoding of vision-language models (ECCV 2024 · better on 1 of 1 shared result)
- LLaVA-UHD: an LMM Perceiving any Aspect Ratio and High-Resolution Images (ECCV 2024 · better on 1 of 1 shared result)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (ICML 2024 · similarity 0.94)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.90)
- Bridging Vision and Language Spaces with Assignment Prediction (ICLR 2024 · similarity 0.90)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.89)
- HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.89)
- Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
