Research map / Vision-language and multimodal / Multimodal LLMs
Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VisInContext, which renders overflow long text as images encoded by a frozen ViT and resampler so Flamingo-style MLLMs can see far longer in-context text at lower token cost.
Problem
Multimodal models trained on interleaved documents are limited to short in-context text (e.g., 256 tokens) because of GPU memory and compute costs of long text sequences.
Approach
Convert omitted long text segments into rendered RGB text images and encode them with the shared frozen vision encoder and a learnable resampler, using token masking and a text-centric contrastive loss (TCCL) to align visual text tokens with LLM text embeddings, thereby extending in-context text length without extra LLM tokens.
Key results
- With 56B MOE, 32-shot mean across benchmarks rises from 55.8% (256 ICL tokens) to 57.8% (2048 ICL tokens) with VisInContext.
- On OpenFlamingo-9B, DocVQA test improves from 48.2 to 52.2 and OCR VQA from 51.5 to 58.4 with VisInContext.
- Text-only 32-shot with rendered text images uses 10 text tokens versus 426 for raw text, with mean 33.6 vs 34.4 for the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| render text as images (VisInContext) to extend ICL from 256 to 2048 tokens | Open-Flamingo MOE 56B · few-shot VL benchmarks (32-shot) | Mean: 55.8 -> 57.8 | Helped |
| render text as images instead of raw text tokens | Open-Flamingo 9B · text-only in-context few-shot (32-shot) | Mean: 34.4 -> 33.6 with 10 vs 426 text tokens | Mixed |
| render text as images for pretraining | Open-Flamingo 9B · document understanding (DocVQA test, OCR VQA) | ANLS / accuracy: DocVQA test 48.2 -> 52.2; OCR VQA 51.5 -> 58.4 | Helped |
| rendered text image pretraining | FuYu9B · DocVQA val | ANLS: 42.3 -> 44.5 | Helped |
| surrounding text rendered as image instead of raw text | sequential multimodal retrieval model · OBELICS-Hybrid6 | Seq-I: 18.9 -> 22.7 | Helped |
| surrounding text rendered as image instead of raw text | sequential multimodal retrieval model · OBELICS-Hybrid6 | Seq-T: 67.5 -> 66.5 | No clear effect |
| remove token masking (text image + TCCL only) (ablation) | VisInContext · text-only 4-shot ablation | TextVQA: 21.8 -> 18.3 (token masking removed, TCCL removed in same row set) | Hurt |
| add token masking (ablation) | text image pipeline · text-only 4-shot ablation | TextVQA: 15.0 -> 18.3 (text image without vs with token masking, TCCL off) | Helped |
| add TCCL on top of token masking (ablation) | VisInContext · text-only 4-shot ablation | TextVQA: 18.3 -> 21.8 | Helped |
| add TCCL without token masking (ablation) | VisInContext · text-only 4-shot ablation | TextVQA: 15.0 -> 15.3 (text image + TCCL vs no text image baseline 15.3) | No clear effect |
| font size increased from 10 to 12 (or lowered to 8) for DocVQA (ablation) | VisInContext text rendering · DocVQA val | ANLS: best at font 8 (45.6) vs 44.3 at font 10 and 36.2 at font 12 | Mixed |
| larger font interval threshold for text rendering (ablation) | VisInContext text rendering · TextVQA and DocVQA val | TextVQA / DocVQA: TextVQA 21.8 at threshold 2 -> 15.3 at threshold 32; DocVQA 44.3 -> 36.6 | Hurt |
Surprising findings
- Token masking of raw image tokens was necessary; without it the model performed about the same as the baseline, so the model otherwise ignored rendered text images.
Limitations
- Requires processing a fixed-size image even for brief texts.
- Few-shot performance with rendered text is slightly below the raw-text baseline in the text-only setting (34.4 vs 33.6 mean).
- Exploratory study; does not present state-of-the-art models.
Experimental setup
Datasets: DataComp1B (180M subset) MMC4 OBELICS Rendered Text OCR data DocVQA OCR VQA OK-VQA TextVQA VizWiz VQAv2 COCO Flickr30K
Benchmarks: OK-VQA TextVQA VizWiz VQAv2 COCO Flickr30K DocVQA OCR VQA Sequential multimodal retrieval (OBELICS-Hybrid6)
Metrics: accuracy CIDEr ANLS retrieval recall
Baselines: OpenFlamingo-9B Open-Flamingo MOE (56B) FuYu-8B
Builds on: Flamingo OpenFlamingo CosMo Pixel
Closest papers by meaning
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.90)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.89)
- Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.89)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.89)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.89)
- LongVILA: Scaling Long-Context Visual Language Models for Long Videos (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.