Research map / Vision-language and multimodal / Human and embodied modeling
Image as a Language: Revisiting Scene Text Recognition via Balanced, Unified and Synchronized Vision-Language Reasoning Network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BUSNet, a scene text recognition network that treats images as language via balanced concatenation, masked modality modeling, and a synchronized vision-language reasoning module with iterative reasoning.
Problem
Scene text recognition models over-rely on either vision or language, and existing designs separate internal/external modeling and decoding schemes, making joint vision-language reasoning hard.
Approach
Concatenate visual and linguistic token sequences along the length dimension to treat images as noisy text, learn a shared-weight ensemble of vision-only, language-only and vision-language predictors via masked modality modeling, and decode all characters in parallel with a VLRM using a special attention mask, optionally refining the text input iteratively.
Key results
- Trained on synthetic data, BUSNet-B reaches 98.3 on IC13 and 91.8 average-style accuracy over six benchmarks (reported Avg 91.8), outperforming ABINet by 0.9% on IC13 and 2.1% on CUTE.
- Trained on real data, BUSNet-B achieves 98.3 IC13, 98.5 SVT, 98.5 IIIT5k, 91.3 SVTP and 96.3 average, with ArT/COCO/Uber accuracies of 83.4/79.4/83.2 on challenging datasets.
- Ablation: the full VL output (98.3 IC13) beats the vision-only output without balanced language training by 2.2% and the language-only output without balanced vision by 36.5% on IC13.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BUSNet rather than PARSeq with synthetic training data | Scene text recognition · ArT, COCO-Text, Uber-Text | accuracy: 70.3 vs 69.1; 65.4 vs 60.2; 43.1 vs 39.9 | Helped |
| Use BUSNet rather than PARSeq with real training data | Scene text recognition · ArT, Uber-Text | accuracy: 83.4 vs 83.0; 83.2 vs 82.4 | Helped |
| Add balanced language features during training (ablation) | BUSNet single-vision output · IC13 | accuracy: 96.7 -> 97.9 | Helped |
| Add balanced vision features during training (ablation) | BUSNet single-language output · IC13 | accuracy: 67.7 -> 70.3 | Helped |
| Add masked modality modeling (ablation) | BUSNet single-vision output · IC13 | accuracy: 2.5 -> 97.9 | Helped |
| Add masked modality modeling (ablation) | BUSNet single-language output · IC13 | accuracy: 0.0 -> 70.3 | Helped |
| Add masked modality modeling (ablation) | BUSNet vision-language output · IC13 | accuracy: 97.1 -> 98.3 | Helped |
| Remove vision-modality masking (ablation) | BUSNet vision-language output · IC13 | accuracy: 98.3 -> 97.6 | Hurt |
| Remove language-modality masking (ablation) | BUSNet vision-language output · IC13 | accuracy: 98.3 -> 97.7 | Hurt |
| Remove both modality masks (ablation) | BUSNet vision-language output · IC13 | accuracy: 98.3 -> 97.1 | Hurt |
| Remove the VLRM attention mask (ablation) | BUSNet vision-language output · IC13 | accuracy: 98.3 -> 96.4 | Hurt |
| Increase VLRM blocks from 6 to 8 (ablation) | BUSNet · IC13 | accuracy: 98.3 -> 98.1 | Mixed |
Surprising findings
- Without the attention mask in VLRM, the language-only output accuracy collapses to vision-only level, suggesting the module degrades to pure vision reasoning.
Limitations
- Struggles with characters with superscripts and non-dictionary words where semantic reasoning is limited (qualitative error analysis).
- Does not achieve best on IIIT5k when trained on real data.
Experimental setup
Datasets: MJSynth SynthText Real training data from Bautista and Atienza 2022
Benchmarks: ICDAR 2013 ICDAR 2015 IIIT5k-Words Street View Text SVT-Perspective CUTE80 COCO-Text ArT Uber-Text
Metrics: word accuracy parameters (M) inference time (ms)
Baselines: SRN RobustScanner TextScanner PREN2D VisionLAN ViTSTR TRBA ABINet SGBANet LevOCR PARSeq MATRN
Builds on: ViT ABINet PARSeq Transformer
Closest papers by meaning
- Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models (ICLR 2026 · similarity 0.89)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
- Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition (CVPR 2025 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.88)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.