Research map / Vision-language and multimodal / Multimodal LLMs
TRINS: Towards Multimodal Language Models that Can Read
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TRINS, a human- and machine-annotated text-rich image instruction dataset built from LAION, and LaRA, a LLaVA-style model with OCR tokens as extra input that is fine-tuned on it to improve text-rich image understanding.
Problem
Multimodal LLMs struggle to read and understand text embedded in images, largely because training data is dominated by natural images with little text.
Approach
A semi-automatic annotation pipeline (CLIP/DiT filtering of LAION, human-written detailed captions with OCR and RAM tags, GPT-4/Llama-70B generated QA) yields a text-rich instruction dataset, and LaRA adds OCR words as text input to a frozen CLIP-ViT-L/14-336 + Vicuna-1.5-13B LLaVA-style model.
Key results
- LaRA achieves 62.8 extract accuracy and 186.6 CIDEr on TRINS-VQA, above the fine-tuned LLaVAR (61.2 and 104.7).
- On TRINS-Cap captioning, LaRA reaches 37.7 BLEU@1 and 21.9 METEOR-style scores range, with CIDEr 46.7 versus 35.7 for fine-tuned LLaVAR.
- On OCRBench, LaRA scores 548 total, behind Gemini (651) and GPT-4V (636) but ahead of Monkey (514) and LLaVA1.5-13B (331).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OCR words as input to LaRA (LLaVAR w/ OCR vs LLaVAR) | LLaVAR zero-shot · TRINS-VQA (extract) | Accuracy: 51.7 -> 58.1 | Helped |
| Fine-tune on TRINS-VQA / instruction data versus zero-shot | LLaVAR · TRINS-VQA (abstract) | CIDEr: 48.2 (LLaVA zero-shot) -> 104.7 (finetuned); LLaVAR 84.9 -> 104.7 | Helped |
| Fine-tune LaRA on TRINS (versus LLaVAR fine-tuned on TRINS) | LLaVA-based model with OCR input · TRINS-VQA (abstract) | CIDEr: 104.7 -> 186.6 | Helped |
| Fine-tune on TRINS-Cap human captions (LaRA vs LaRA zero-shot) (ablation) | LaRA · TRINS-Cap | CIDEr: 21.3 -> 46.7 | Helped |
| Use higher-resolution visual encoder (448) without OCR | mPLUG-Owl2 and Qwen-VL zero-shot · TRINS-VQA (extract) | Accuracy: 61.0 and 63.6 vs LLaVA-style 336 models 23.7 to 51.7 | Mixed |
| Fine-tune LLaVAR on TRINS (general VQA comparison with LLaVA) | LaRA vs LLaVA on general benchmarks · OKVQA/GQA/VSR/VizWiz | accuracy: VizWiz 45.0 -> 53.1; GQA 41.3 -> 42.4; VSR 51.2 -> 53.0; OKVQA 57.8 -> 58.1 | Helped |
| TextDiffuser vs ControlNet and DeepFloyd on hard text rendering | text-to-image generation on TRINS-Gen easy · TRINS-Gen easy | OCR Accuracy: 0.5027 vs 0.4241 (ControlNet) and 0.2192 (DeepFloyd) | Helped |
| Increase OCR word count per image in generation prompts (easy to difficult TRINS-Gen) | text-to-image models (TextDiffuser) · TRINS-Gen difficult | OCR accuracy: 0.5027 (easy) -> 0.0952 (difficult) | Hurt |
Surprising findings
- Abstract answers in TRINS-VQA average 23.9 words, far longer than related datasets (under 4 words).
- Fine-tuning on text-rich data did not degrade natural-image benchmarks; LaRA improved over LLaVA on OKVQA, GQA, VSR and VizWiz.
Limitations
- OCR-free visual encoding still loses text recognition detail, so the model relies on an external OCR tool with its noise.
- Text-to-image models fail to render many words, so the TRINS-Gen difficult set remains unsolved.
- QA data is used directly from LLM outputs without human verification.
Experimental setup
Datasets: LAION-5B RVL-CDIP TRINS LLaVA instruction data TextCaps COCO OCR-VQA TextVQA DocVQA InfoVQA ChartQA
Benchmarks: TRINS-VQA TRINS-Cap TRINS-Gen TRINS-VQA-Human OCRBench OKVQA GQA VSR VizWiz ST-VQA OCR-VQA TextVQA
Metrics: accuracy BLEU METEOR ROUGE CIDEr FID CLIP score OCR accuracy
Baselines: LLaVA LLaVAR LLaVA 1.5 mPLUG-Owl2 Qwen-VL InstructBLIP MiniGPT-4 MiniGPT-v2 Monkey GPT-4V Gemini SD-1.5
Builds on: LLaVA LLaVAR Vicuna CLIP TextDiffuser
Closest papers by meaning
- MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.90)
- ScImage: How good are multimodal large language models at scientific text-to-image generation? (ICLR 2025 · similarity 0.88)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.88)
- OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text (ICLR 2025 · similarity 0.88)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.88)
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.