Research map / Vision-language and multimodal / Multimodal LLMs
MM-IFEngine: Towards Multimodal Instruction Following
Shengyuan Ding, Shenxi Wu, Xiangyu Zhao, Yuhang Zang, Haodong Duan, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Dahua Lin, Jiaqi Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with MM-IFDPO-23k (DPO) | Qwen2-VL-7B-Instruct · MM-IFEval | accuracy: 42.0 -> 52.2 | Improved |
| fine-tune with MM-IFInstruct-23k (SFT) | Qwen2-VL-7B-Instruct · MM-IFEval + MIA + IFEval average | average: 56.6 -> 64.2 | Improved |
| fine-tune with MM-IFDPO-23k (DPO) | LLaVA-NeXT-Llama3-8B · MM-IFEval + MIA + IFEval average | average: 57.9 -> 69.5 | Improved |
| fine-tune with MM-IFDPO-23k (DPO) | LLaVA-NeXT-Llama3-8B · IFEval (text-only) | accuracy (avg): 50.7 -> 69.1 | Improved |
| use DPO instead of SFT on the MM-IF data | LLaVA-NeXT-Llama3-8B · IFEval (text-only) | accuracy (avg): 56.3 -> 69.1 | Improved |
| fine-tune with MM-IFDPO-23k (DPO) | Qwen2-VL-7B-Instruct · VQA benchmarks average (8 benchmarks) | average: 72.3 -> 72.4 | No clear effect |
| fine-tune with MM-IFInstruct-23k (SFT) | Qwen2-VL-7B-Instruct · VQA benchmarks average (8 benchmarks) | average: 72.3 -> 70.7 | Hurt |
| fine-tune with MM-IFDPO-23k (DPO) | LLaVA-NeXT-Llama3-8B · MM-IFEval perception-level | accuracy: 21.0 -> 21.0 | No clear effect |
| remove 100% of constraints for rejected DPO responses instead of 33% (ablation) | Qwen2-VL-7B-Instruct · MM-IFEval + MIA + IFEval average | average: 65.8 -> 66.7 | Improved |
| remove 100% of constraints for rejected DPO responses instead of 33% (ablation) | LLaVA-NeXT-Llama3-8B · MM-IFEval + MIA + IFEval average | average: 67.3 -> 69.5 | Improved |
| generate DPO rejected responses with the image present instead of without image (ablation) | Qwen2-VL-7B-Instruct · MM-IFEval + MIA + IFEval average | average: 63.4 -> 66.7 | Improved |
| generate DPO rejected responses with the image present instead of without image (ablation) | LLaVA-NeXT-Llama3-8B · MM-IFEval | accuracy: 44.7 -> 49.3 | Improved |
Closest papers by meaning
- MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs (ICLR 2025 · similarity 0.91)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.90)
- Re-Imagining Multimodal Instruction Tuning: A Representation View (ICLR 2025 · similarity 0.90)
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.89)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.89)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
