Research map / Vision-language and multimodal / Multimodal LLMs

MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MergeMix, a token-merge-based mixup augmentation for image classification and MLLM alignment, which builds mixed-image losers and optimizes a mixing-ratio-weighted SimPO loss alongside SFT.

Problem

Preference alignment of MLLMs relies on costly human annotations (SFT) or unstable, compute-heavy RL; existing preference-pair construction via random augmentation yields poorly controlled losers. The paper asks whether classical mixup can build controllable preference pairs for MLLM post-training.

Approach

Generate mixed images using attention maps recovered from token merging (ToMe bipartite soft matching) to create spatially contextual masks, re-scale the mixing ratio with a Gaussian-based policy, and use mixed images as losers with the mixing ratio as a soft margin in a SimPO-style ranking loss added to SFT.

Key results

• On CIFAR-100 with DeiT-Tiny (200 epochs), MergeMix reaches 77.46% top-1 vs 75.31% for TransMix, a gain of about 2.15 points.
• On LLaVA-v1.5-7B with full vision tokens, MergeMix averages 66.40 vs 65.57 for the LLaVA baseline (+0.83), and 66.45 at 288 tokens.
• On Qwen2.5-VL-7B-Instruct, MergeMix improves the 9-benchmark average from 61.19 to 64.07 (+2.88) over the official checkpoint.

Surprising findings

• ResizeMix degrades LLaVA average accuracy (-1.44 and -2.24 points) relative to SFT, showing that not all mixup variants transfer to MLLMs.

Limitations

• Augmentation only perturbs the image modality; text is untouched.
• The token-merging policy is static and not learned.
• Gains on MLLM benchmarks are modest (often under 1 point on LLaVA) and mixed across benchmarks; some benchmarks regress (e.g., POPE, MMBench-CN in some settings).

Experimental setup

Datasets: CIFAR-100 ImageNet-1K CUB-200 FGVC-Aircraft Stanford-Cars LLaVA-v1.5-mix665k LCS-558K ESC-50 UrbanSound8K

Benchmarks: CIFAR-100 ImageNet-1K CUB-200 FGVC-Aircraft Stanford-Cars VQAv2 GQA VizWiz ScienceVQA TextVQA MME MMBench

Metrics: top-1 accuracy throughput FLOPs expected calibration error average accuracy

Baselines: MixUp CutMix ResizeMix TransMix SMMix MixPro DeiT AutoMix PuzzleMix LLaVA-NeXT SeVa VisionThink

Builds on: Mixup CutMix ToMe SimPO DPO LLaVA-v1.5

Closest papers by meaning

• Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment (ICLR 2025 · similarity 0.89)
• Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment (NeurIPS 2025 · similarity 0.88)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
• Linguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal Alignment (CVPR 2024 · similarity 0.88)
• HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.88)
• Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning (ICLR 2026 · similarity 0.88)