AI Paper Atlas

Research map / Vision-language and multimodal / Visual question answering

Towards Models that Can See and Read

Roy Ganz, Oren Nuriel, Aviad Aberdam, Yair Kittenplon, Shai Mazor, Ron Litman

ICCV 2023 · Vision-language and multimodal › Visual question answering · approach: Vision-language model VQA evaluation · 15 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add OCR tokens to the existing text encoder naively (ablation)BLIP · TextVQA val (Comb.)accuracy: 32.43 -> 43.02Improved
add designated OCR encoder with 2-D info (ablation)UniTNT_BLIP · TextVQA val (Comb.)accuracy: 43.02 -> 46.13Improved
add VL-OCR decoding fusion (ablation)UniTNT_BLIP · TextVQA val (Comb.)accuracy: 46.13 -> 47.38Improved
add OCR language-modeling loss L_OCR-LM (ablation)UniTNT_BLIP · TextVQA val (Comb.)accuracy: 47.38 -> 52.66Improved
add OCR-BC loss with 2-D information (ablation)UniTNT_BLIP · TextVQA val (Comb.)accuracy: 52.66 -> 55.21Improved
freeze the vision encoder (ablation)UniTNT_BLIP · TextVQA valaccuracy: 52.45 -> 55.21Improved
add scene-text understanding (UniTNT) with VQAv2-only trainingBLIP · VQAv2 and TextVQAaccuracy: +3.19% VQA, +12.16% TextVQAImproved
add scene-text understanding (UniTNT) with TextVQA-only trainingBLIP · VQAv2 and TextVQAaccuracy: +19.67% TextVQA, -3.15% VQAMixed
add UniTNT with TextCaps-only finetuningBLIP · TextCaps vs COCO CaptionsCIDEr: +19.4 TextCaps test, -14.4 COCOMixed
combined training on vision and scene-text data rather than separate finetuning (ablation)UniTNT_BLIP · VQAaverage accuracy: 58.01 -> 67.56Improved
combined training instead of TextCaps-only finetuning (ablation)UniTNT_BLIP · TextCaps valCIDEr: 130.5 -> 119.1Hurt
add UniTNT to BLIP in Caps-only trainingBLIP · COCO Karpathy-testCIDEr: 133.3 -> 133.7Improved

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy