Research map / Vision-language and multimodal / Human and embodied modeling
UNIT: Unifying Image and Text Recognition in One Vision Encoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UNIT, a training framework that upgrades an image-pretrained ViT (OpenCLIP ViT-H/14) to also do OCR, using a lightweight OPT-125M language decoder and a vision decoder that distills the original encoder's features, trained in intra-scale and inter-scale stages.
Problem
Vision encoders like ViTs excel at image recognition but cannot simultaneously handle text recognition (OCR, document QA), and existing document-specific or ensemble approaches either discard image capability or add inference cost.
Approach
Attach a lightweight OPT-125M language decoder for autoregressive OCR and a two-layer MLP vision decoder that reconstructs the original ViT's token features on natural images, training at intra-scale resolutions and then at exchanged (scale-swapped) resolutions so the single unchanged encoder handles both images and documents with no added inference cost.
Key results
- On the in-house SYN-L-val OCR set, UNIT reaches 95.33 F1 vs 91.20 for Vary* and 93.90 for RADIO* at 896x896 (Table 1), and 67.14/41.48/58.87 F1 on FUNSD/SROIE/CORD vs Donut's 9.08/8.94/16.64.
- UNIT achieves 78.76% zero-shot ImageNet-1K accuracy and 50.19 ADE20k mIoU, edging out RADIO-H/14 (78.62, 49.01) with the same 632M size.
- In LVLM naive-resolution setting, UNIT scores 43.0 DocQA, 46.0 ChartQA and 28.7 InfoVQA vs RADIO-H's 18.2/20.2/24.5, while VQAv2 stays comparable (72.3 vs 72.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the image captioning task during intra-scale pretraining (ablation) | UNIT · ImageNet-1K | zero-shot classification accuracy: 76.24% to 78.54% | Helped |
| use feature reconstruction loss weight λ=2 rather than λ=1, with image captioning (ablation) | UNIT · ImageNet-1K | zero-shot classification accuracy: 75.16 to 78.54 | Helped |
| use feature reconstruction loss weight λ=2 rather than λ=1, with image captioning (ablation) | UNIT · SYN-L-val | OCR F1: 90.88 to 94.57 | Helped |
| add inter-scale finetuning (ablation) | UNIT · ImageNet-1K at exchanged resolution | zero-shot classification accuracy: 0.40 to 80.49 | Helped |
| add inter-scale finetuning (ablation) | UNIT · SYN-S-val | OCR F1: 34.99 to 92.68 | Helped |
| use random feature sampling during inter-scale finetuning (ablation) | UNIT · ImageNet-1K at exchanged resolution | zero-shot classification accuracy: 31.66 to 80.49 | Helped |
| use random feature sampling during inter-scale finetuning (ablation) | UNIT · SYN-S-val | OCR F1: 92.40 to 92.68 | Helped |
| train with λ=0 for feature reconstruction loss (ablation) | UNIT without image captioning · ImageNet-1K | zero-shot classification accuracy | Hurt |
Surprising findings
- Disabling the feature reconstruction loss (lambda=0) makes zero-shot classification fail entirely (N/A), because the vision encoder loses alignment with the OpenCLIP text encoder.
- Without random feature sampling, zero-shot classification at the exchanged resolution collapses to 31.66% (vs 80.49% with it), showing gradient overflow issues with high-resolution natural images.
Limitations
- Inter-scale finetuning on scale-exchanged data is needed; without it, OCR at exchanged resolution drops to 34.99 F1.
- Several baselines (Vary, RADIO) were retrained by the authors on their own data, and the in-house SYN and MD evaluation sets are self-constructed, which limits comparability.
- Markdown conversion underperforms Nougat (78.50 vs 86.71 F1), which was trained on markdown data.
Experimental setup
Datasets: Conceptual Captions ShareGPT4V LAION-COCO LLaVA-80k LLaVA-CC665k DocVQA ChartQA Nougat markdown data synthetic arXiv PDF OCR data
Benchmarks: FUNSD SROIE CORD SYN-L-val MD-val ImageNet-1K ADE20k VQAv2 GQA OKVQA ChartQA DocVQA
Metrics: F1 top-1 accuracy mIoU accuracy OCRBench score MME score
Baselines: Donut Nougat Vary RADIO EfficientViT-L1 SwinV2-S ConvNext-B MViTV2-B NFNet-F3 MaxViT-B OpenCLIP-H/14 RADIO-L/14
Builds on: Vision Transformer OpenCLIP LLaVA-1.5 Cropped Position Embedding Q-Former
Closest papers by meaning
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.85)
- Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models (ICLR 2026 · similarity 0.85)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.84)
- EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE (AAAI 2024 · similarity 0.84)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.