Research map / Vision-language and multimodal / Cross-modal learning

LoMo: Local Modality Substitution for Deeper Vision-Language Fusion

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LoMo, a data curation method that rewrites part of text-only SFT samples into text–image–text interleaved sequences by rendering a middle text span as an image (with perceptual distortion), improving VLM performance and cross-carrier alignment without architectural changes.

Problem

VLMs are carrier-sensitive: replacing a text question with its rendered-image counterpart causes large accuracy drops, attributed to asymmetric text/image roles in multimodal training corpora.

Approach

Reformulate text-only instructions by rendering a selected middle span as an image (with LaTeX-aware routing and perceptual distortions) and embedding it between surrounding text, so standard SFT implicitly enforces cross-carrier representational invariance.

Key results

• LoMo improves average standard-evaluation accuracy over Standard SFT by +2.68 on LLaVA-OV1.5-8B (40.88 to 43.56) and +2.82 on Qwen3.5-9B (54.43 to 57.25) across 13 benchmarks.
• Under rendered evaluation, LoMo gains +18.86 average on LLaVA-OV1.5-8B and +11.92 on Qwen3.5-9B, and compresses Qwen3.5-9B's standard-to-rendered drop from 11.17 to 2.07 points.
• Pairwise cross-modal distance under LoMo drops to 0.49 at 4M while Standard SFT rises from 0.52 to 0.57; LoMo reduces average distance by 14.2% vs Standard SFT.

Surprising findings

• Standard SFT increases paired text-vs-rendered-image distance during training (0.52 to 0.57), while LoMo decreases it.
• Naively rendering the full question as an image yields only +1.19 average gain, far less than span-localized LoMo.

Limitations

• LoMo applied only during SFT; not tested with pretraining or RL post-training.
• Span localization uses a block-level middle-span heuristic rather than difficulty-aware selection.
• Validated only on two 8B-9B backbones.

Experimental setup

Datasets: LLaVA-OneVision-1.5 SFT corpus

Benchmarks: MMMU MMMU-Pro MathVista ZeroBench WeMath SimpleVQA HallusionBench MM-IFEval MMLongBench-Doc DocVQA CC-OCR V*

Metrics: accuracy

Baselines: Standard SFT Full-Text Rendering

Builds on: LLaVA-OneVision-1.5 Modality Integration Rate

Closest papers by meaning

• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.91)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.91)
• The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.91)
• Remodeling Semantic Relationships in Vision-Language Fine-Tuning (AAAI 2026 · similarity 0.91)
• DiM$^3$: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging (NeurIPS 2026 · similarity 0.90)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.90)