AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document Understanding

Ofir Abramovich, Niv Nayman, Sharon Fogel, Inbal Lavi, Ron Litman, Shahar Tsiper, Royee Tichauer, Srikar Appalaraju, Shai Mazor, R. Manmatha

ECCV 2024 · Vision-language and multimodal › Human and embodied modeling · approach: Layout-aware document understanding · 9 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace Swin patch-merging with ViLMA prompt-aware layers (ablation)VisFocus-B · ChartQArelaxed accuracy: 52.5 -> 54.7Improved
replace Swin patch-merging with ViLMA prompt-aware layers (ablation)VisFocus-B · DocVQA valANLS: 70.9 -> 71.3Improved
add LMPM pre-training stage (ablation)VisFocus-B · DocVQA valANLS: 71.3 -> 71.8Improved
add LMPM pre-training stage (ablation)VisFocus-B · ChartQArelaxed accuracy: 54.7 -> 55.7Improved
randomly skip concatenating the prompt to the LM input (ablation)VisFocus-B · ChartQArelaxed accuracy: 55.7 -> 57.1Improved
render prompt onto the image (Pix2Struct style) instead of LM-only prompt (ablation)baseline · DocVQA / ChartQAANLS / relaxed accuracyHurt
use ViLMA in all blocks instead of only the last two (ablation)VisFocus-B · ChartQArelaxed accuracy gain over baseline: +2.8 -> +4.6Improved
use T5-Base encoder rather than independent token embeddings for the prompt (ablation)VisFocus-B · ChartQArelaxed accuracy: 55.4 -> 57.1Improved
use ViLMA prompt guidance on denser documentsVisFocus-B · DocVQA val (documents with at least 400 to 800 words)ANLS gain over baseline: +0.7 -> +2.3Improved
apply VisFocus-S over its baselineVisFocus-S · AI2Dexact match: 42.7 -> 42.6No clear effect
apply VisFocus-B versus Pix2Struct-B on infographicsVisFocus-B · InfoVQAANLS: 38.2 -> 31.9Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy