Research map / Vision-language and multimodal / Multimodal LLMs
Harmonizing Visual Text Comprehension and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TextHarmony, a unified model for visual text comprehension and generation, using Slide-LoRA (dynamically gated modality-specific and shared LoRA experts) and a Gemini-captioned DetailedTextCaps-100K dataset.
Problem
Multimodal generation of text and images within one model degrades performance due to inconsistency between vision and language generation objectives, especially for visual text tasks.
Approach
Slide-LoRA gates between text-specific and image-specific LoRA experts plus a shared LoRA expert inserted in the LLM (and vision encoder), partially decoupling the generation spaces within a single model instance with ~2% added parameters; also a detailed visual-text caption dataset generated with Gemini Pro.
Key results
- TextHarmony with Slide-LoRA improves average comprehension accuracy by 2.5% over the variant without Slide-LoRA (e.g., DocVQA 47.1 vs 44.0, OCRVQA 55.3 vs 51.9).
- On visual text editing, NED rises from 0.68 to 0.75 and CLIP score from 0.33 to 0.35 with Slide-LoRA.
- TextHarmony reaches 88.7% Acc@0.5 on MARIO-Eval text grounding, above TGDoc (82.5) and DocOwl 1.5 (84.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Slide-LoRA (ablation) | TextHarmony · Visual text comprehension tasks (average) | accuracy: +2.5% | Helped |
| add Slide-LoRA (ablation) | TextHarmony · AnyText-benchmark-EN | NED: 0.68 to 0.75; improvement of 0.07 | Helped |
| add Slide-LoRA (ablation) | TextHarmony · MARIO-Eval | CLIP Score: 0.33 to 0.35; 0.02 | Helped |
| vary the number of Slide-LoRA modules from n=3, s=1 to n=9, s=3 (ablation) | TextHarmony · TextVQA | accuracy: 60.2%, 60.4%, and 60.4% | No clear effect |
| insert Slide-LoRA in the vision encoder only rather than both the vision encoder and LLM (ablation) | TextHarmony · TextVQA | accuracy: 60.2 -> 58 | Hurt |
| insert Slide-LoRA in the LLM only rather than both the vision encoder and LLM (ablation) | TextHarmony · TextVQA | accuracy: 60.2 -> 59.9 | Hurt |
| add DetailedTextCaps-100K (ablation) | TextHarmony · AnyText-benchmark-EN | NED: 0.70 to 0.75 | Helped |
| add DetailedTextCaps-100K (ablation) | TextHarmony · MARIO-Eval | FID: 368 to 342 | Helped |
| add DetailedTextCaps-100K (ablation) | TextHarmony · MARIO-Eval | CLIP Score: 0.32 to 0.35 | Helped |
| use TextHarmony-Chat trained only with visual comprehension data | TextHarmony · Visual text comprehension tasks | accuracy: 3.96% to 1.5% | Mixed |
Surprising findings
- Inserting Slide-LoRA only into the vision encoder substantially degrades performance, while inserting only into the LLM causes limited degradation.
- The number of LoRA modules (n=3 to 9) has little effect on performance.
Limitations
- Visual text perception and comprehension lag behind some state-of-the-art open-source models.
- Image generation is less robust for dense-text scenarios.
- Reported comparison against GPT-4V judgment may contain biases; small sample (88 pairs).
Experimental setup
Datasets: MARIO-LAION DocStruct4M Monkey VQA training set DetailedTextCaps-100K
Benchmarks: InfoVQA DocVQA ChartQA TabFact WTQ TextVQA OCRVQA STVQA OCRBench MARIO-Eval AnyText-benchmark-EN
Metrics: accuracy NED FID CLIP Score Acc@0.5
Baselines: Monkey InternLM-XComposer2 InternVL mPLUG-Owl2 LLaVA1.5-7B SEED-LLaMA-14B MiniGPT-5 MM-Interleaved AnyText TextDiffuser-2 GlyphControl TGDoc
Builds on: MM-Interleaved LoRA Monkey AnyText
Closest papers by meaning
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.90)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.89)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.89)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.88)
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.88)
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.