Research map / Vision-language and multimodal / Human and embodied modeling
Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a regular-sized scene text recognizer (E2STR) with an in-context training strategy (Split and Transform) so it can adapt to new scenarios training-free by retrieving visually similar in-context prompts at inference.
Problem
Scene text recognition models degrade across diverse domains and fine-tuning per scenario is costly; in-context learning does not emerge in STR models trained on randomly concatenated samples, and M-LLMs are too expensive as recognizers.
Approach
Generate context-rich scene text sequences by splitting character-annotated images into overlapping sub-samples and transforming images (augmentation) to create inter-connected samples for training, then retrieve kNN visual-similarity prompts at inference to enable training-free in-context recognition.
Key results
- E2STR-ICL reaches 91.33% average word accuracy over 12 common benchmarks, 1.08% above non-context E2STR-base and 0.83% above MAERec.
- On three cross-domain datasets, training-free E2STR-ICL averages 78.17%, 4.03% above fine-tuned-free MAERec (74.14%) and above fine-tuned MAERec by 1.16%.
- On hard case rectification with 10-20% annotated hard cases, E2STR-ICL reduces the error rate of remaining hard cases by up to 32%, versus at most 2.23% for fine-tuned MAERec.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add in-context prediction with selected prompts | E2STR · 12 common benchmarks (average) | word accuracy: 90.25 -> 91.33 | Helped |
| add in-context prediction with selected prompts | E2STR · MPSC | word accuracy: 81.26 -> 83.64 | Helped |
| add in-context prediction with selected prompts | E2STR · EIST | word accuracy: 69.66 -> 76.77 | Helped |
| add in-context prediction with selected prompts | E2STR · IAM | word accuracy: 69.51 -> 74.10 | Helped |
| use context-rich training sequences with the Transform and Split Strategies (ablation) | E2STR · EIST | word accuracy: 26.82 -> 76.77 | Helped |
| add the Transform Strategy to vanilla STR training (ablation) | E2STR · EIST | word accuracy: 26.82 -> 75.60 | Helped |
| add the Split Strategy to vanilla STR training (ablation) | E2STR · EIST | word accuracy: 26.82 -> 73.09 | Helped |
| add the Transform Strategy and Split Strategy (ablation) | E2STR · EIST non-context prediction | word accuracy: 69.69 -> 69.66 | No clear effect |
| replace random prompt selection with kNN visual-similarity selection (ablation) | E2STR-ICL · EIST | word accuracy: 70.65 -> 76.77 | Helped |
| increase the in-context pool size (ablation) | E2STR-ICL · IAM, EIST, and MPSC | word accuracy | Helped |
| increase the number of in-context prompts (ablation) | E2STR-ICL · HOST, Total Text, and WordArt | word accuracy | Helped |
| use one in-context prompt rather than none (ablation) | E2STR-ICL · HOST | word accuracy: 73.30 -> 74.34 | Helped |
Surprising findings
- Naively adding in-context prompts to a model trained on randomly concatenated scene text drops accuracy sharply (69.69% to 26.82% on EIST), showing ICL fails without context-rich training.
Limitations
- Can erroneously rectify correct predictions when prompts are misleading (rare).
- Cannot recognize characters not included in the lexicon.
- Gains depend on the in-context pool containing relevant examples.
Experimental setup
Datasets: Union14M-L 600k synthesized scene text images EIST
Benchmarks: IIIT5k SVT IC13 IC15 SVTP CUTE80 COCO Text CTW Total Text HOST WOST WordArt
Metrics: word accuracy
Baselines: ASTER NRTR SAR SATRN ABINet PARSeq MAERec
Builds on: Flamingo-style cross-attention interleaved training MAERec Vision Transformer (MAE pretraining) OPT
Closest papers by meaning
- Recognition-Synergistic Scene Text Editing (CVPR 2025 · similarity 0.84)
- EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering (CVPR 2025 · similarity 0.81)
- ConText: Driving In-context Learning for Text Removal and Segmentation (ICML 2025 · similarity 0.81)
- ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.81)
- ContextNav: Towards Agentic Multimodal In-Context Learning (ICLR 2026 · similarity 0.80)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.