Research map / Vision-language and multimodal / Multimodal LLMs

DAVE: A VLM Vision Encoder for Document Understanding and Web Agents

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DAVE, a document/web-specialized vision encoder trained via MAE self-supervision, supervised autoregressive pretraining, learned weight-merging across decoders, and ensembling with frozen SigLIP2, evaluated in LLaVA-style VLMs on document, VQA, web grounding and Mind2Web.

Problem

General VLM vision encoders (e.g., SigLIP2) lack the low-level structural and spatial features needed for document understanding and web agents, and annotated document/web data is scarce.

Approach

Combine MAE-style self-supervised pretraining on 20M unlabeled document/web images (with raw-pixel reconstruction to avoid divergence) with supervised autoregressive pretraining across multiple text decoders, merge the resulting encoders via learned coefficients distilled from teacher features, and concatenate with frozen SigLIP2 features.

Key results

• On Llama-3.2-3B, DAVE beats the strongest baseline SigLIP 2 on 8 document/web benchmarks by an average of 10.5% (e.g., DocVQA 82.1 vs 72.1).
• On Mind2Web cross-task step SR, DAVE reaches 26.1 vs 19.6 for the best baseline Dolphin.
• On DocLayNet, DAVE scores 74.1 mAP vs 70.8 for SigLIP 2; on DocBank it scores 56.9 vs 51.7.

Surprising findings

• Normalized-pixel MAE objective diverged on document and web screenshots, motivating a switch to raw-pixel reconstruction.
• Specialized document encoders (Pix2Struct, Dolphin) underperform general SigLIP 2 and AIMv2 on several VLM benchmarks when used alone.

Limitations

• Operates at a fixed resolution
• Depends on an external pretrained generalist encoder (SigLIP 2)
• Requires substantial compute for self-supervised training; encoder is not conditioned on prior agent actions

Experimental setup

Datasets: DocFM Common Crawl web screenshots (Common-Web) PlotQA ChartQA FinTabNet DaTikZ PubTables UGround PaddleOCR-processed arXiv PDFs LLaVA-1.5-mix 665K DocVQA AI2D

Benchmarks: AI2D OCRBench DocVQA InfoVQA ChartQA MMMU RealWorldQA TextVQA VisualWebBench Screenspot-V2 WebSRC Mind2Web

Metrics: accuracy mAP element accuracy step success rate center accuracy

Baselines: SigLIP 2 AIMv2 DinoV2 MAE-Web Pix2Struct Dolphin DiT

Builds on: Masked Autoencoder SigLIP 2 LLaVA-NeXT Fisher Merge Granite Vision

Closest papers by meaning

• DocFormerv2: Local Features for Document Understanding (AAAI 2024 · similarity 0.88)
• DocVLM: Make Your VLM an Efficient Reader (CVPR 2025 · similarity 0.86)
• Boosting Document Parsing Efficiency and Performance with Coarse-to-Fine Visual Processing (CVPR 2026 · similarity 0.86)
• Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.85)
• AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.85)
• The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.85)