Research map / Vision-language and multimodal / Human and embodied modeling
Modality-Specialized Synergizers for Interleaved Vision-Language Generalists
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoSS, modality-specialized LoRA adapters (convolutional LoRA for image tokens, linear LoRA for text) added to frozen interleaved VLGs, plus LeafInstruct, a 184,982-instance interleaved instruction-tuning dataset.
Problem
Existing vision-language generalists produce incoherent, low-quality interleaved text and images, partly because a single set of parameters and architecture models both discrete/continuous image features and linear text, and because they lack large-scale interleaved instruction data.
Approach
Attach frozen-backbone LoRA adapters split by modality: a convolutional LoRA (convolution applied in full-dimension feature space with causal top/left padding) for image patch hidden states and a standard linear LoRA for text hidden states, routed by output modality during interleaved generation.
Key results
- Emu2 + MoSS improves average InterleavedEval score by 97.76% over Emu2 and beats open-source GILL by 34.7% on average across five aspects.
- On MagicBrush full test set, Emu2 + MoSS reaches AVG 62.26 vs 60.53 for SEED-X-Edit.
- On MSCOCO-30K, Emu2 + MoSS achieves FID 18.2 vs 23.4 for Emu2 + LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add modality-specialized Linear LoRA for text and Convolutional LoRA for images, trained on LeafInstruct | Emu2 · InterleavedBench | TIC: 1.22 -> 3.54 | Helped |
| Add modality-specialized Linear LoRA for text and Convolutional LoRA for images, trained on LeafInstruct | Chameleon · InterleavedBench | Text Quality: 3.33 -> 2.98 | Hurt |
| Add modality-specialized Linear LoRA for text and Convolutional LoRA for images, trained on LeafInstruct | Chameleon · InterleavedBench | Image Coherence: 0.28 -> 1.05 | Helped |
| Replace shared traditional linear LoRA with MoSS modality-specialized adapters | Emu2 · InterleavedBench | Image Coherence: 1.99 -> 3.41 | Helped |
| Replace separate linear MoE-LoRA experts with Linear LoRA for text and Convolutional LoRA for images | Emu2 · InterleavedBench | TIC: 2.62 -> 3.54 | Helped |
| Replace MoSS-ConvLoRA with SAM-ConvLoRA (ablation) | MoSS · InterleavedBench | Image Coherence: 3.41 -> 3.17 | Hurt |
| Use convolution on full-rank original image features instead of first reducing features to a low-rank space (ablation) | MoSS · InterleavedBench | Perceptual Quality: 3.33 -> 3.62 | Helped |
| Increase LoRA rank from 64 to 128 (ablation) | MoSS · InterleavedBench | Image Coherence: 2.20 -> 2.64 | Helped |
| Increase LoRA rank from 128 to 256 (ablation) | MoSS · InterleavedBench | Image Coherence: 2.64 -> 3.41 | Helped |
| Fine-tune all Emu2 parameters instead of only MoSS adapter parameters (ablation) | Emu2 · InterleavedBench | Image Coherence: 3.41 -> 2.98 | Hurt |
| Fine-tune all Emu2 parameters instead of only MoSS adapter parameters (ablation) | Emu2 · InterleavedBench | Text Quality: 2.61 -> 3.20 | Helped |
| Train on LeafInstruct with MoSS instead of the original Chameleon setup | Chameleon · InterleavedBench human text-quality evaluation | preference wins: 28 wins, 54 ties, 18 losses | Mixed |
Surprising findings
- Chameleon + MoSS scores lower than original Chameleon on LLM-judged text quality, attributed to judge verbosity bias; human evaluation preferred MoSS text (28 wins, 54 ties, 18 losses).
- Full fine-tuning of Emu2 gives better text quality but worse image quality than MoSS, despite training far more parameters.
Limitations
- Chameleon + MoSS shows a text-quality drop under the LLM judge, which the authors attribute to verbosity bias rather than true degradation.
- Evaluation relies largely on GPT-4o-based InterleavedEval scores with a small human study; Chameleon implementation is a third-party reimplementation since original checkpoints are unavailable.
- Generated images from the method can still show distortions and lower perceptual quality on complex tasks such as composition and editing.
Experimental setup
Datasets: LeafInstruct MMC4 MMDialog VIST WikiWeb2M YouCook2 Vision-Flan LAION-COCO InterleavedBench MagicBrush MSCOCO GenEval
Benchmarks: InterleavedBench MagicBrush MMBench MME MMMU Pope MM-Vet MSCOCO-30K GenEval
Metrics: InterleavedEval score (0-5) on Text Quality, Perceptual Qual CLIP-I DINO CLIP-T FID accuracy/score on understanding benchmarks
Baselines: GILL MiniGPT-5 Emu2 Chameleon Gemini 1.5 + SDXL GPT-4o + DALL·E 3 SEED-X-Edit LoRA MoE-LoRA
Builds on: LoRA Mixture-of-Experts LoRA ConvLoRA (SAM-ConvLoRA) Emu2 Chameleon InterleavedEval
Closest papers by meaning
- Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.91)
- SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding (CVPR 2025 · similarity 0.90)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.90)
- Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.89)
- Exploring the Better Multimodal Synergy Strategy for Vision-Language Models (AAAI 2025 · similarity 0.89)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.