Research map / Large language models / Language, safety and interpretability

A Simple yet Effective Layout Token in Large Language Models for Document Understanding

CVPR 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LayTokenLLM, which compresses each OCR text segment's bounding box into a single learnable-query layout token, shares position IDs between text and layout tokens, and pretrains with a next-interleaved text-and-layout token prediction objective for document understanding.

Problem

Layout-as-token methods for LLM document understanding allocate extra position IDs to layout tokens, which reduces the position range available for text and causes untrained position IDs in long multi-page contexts.

Approach

Represent each text segment's layout as one layout token (attention pooling over a projected bounding box with a learnable query), assign it the position ID of the segment's first text token so no extra position IDs are needed, and train with a next-token objective that predicts both text tokens (CE) and layout coordinates (MSE).

Key results

• LayTokenLLM-8B reaches 74.31 ANLS on MP-DocVQA and 52.00 on DUDE, versus 69.42 and 46.77 for MPLUG-DocOwl2-8B among compared MLLMs.
• LayTokenLLM-7B scores 92.03 on SIBR vs 68.39 for InternVL2-8B, while LayTokenLLM-8B achieves 85.11 on DocVQA.
• Ablation: layout tokenizer plus shared position IDs (LayPosID) yields +2.3% on multi-page tasks; NTLP adds 2.2% single-page and 0.8% multi-page ANLS average.

Measured effects reported in the paper

Surprising findings

• Explicit text-with-layout inputs (e.g., '{text, Box}' format) underperform the compressed layout token on multi-page DocVQA, with ANLS gaps of over 10 points among layout-as-token baselines.

Limitations

• May overlook graphical elements such as charts and icons.
• Single layout token compression loses some information (single-page performance slightly degrades without shared position IDs).
• Only fine-grained layout relationship prediction left for future work.

Experimental setup

Datasets: LayoutLLM layout-aware SFT data DocVQA SIBR MP-DocVQA DUDE FUNSD CORD

Benchmarks: DocVQA SIBR FUNSD CORD MP-DocVQA DUDE

Metrics: ANLS F1 FLOPs/MACs

Baselines: Qwen1.5-7B-Chat Llama3-8B-Instruct DocLLM LayoutLLM LayTextLLM InternVL2-8B TextMonkey QwenVL Idefics3-8B MPLUG-DocOwl2-8B LongVA-7B LLaVA-next-interleave-7B

Builds on: LayoutLLM LayTextLLM RoPE LoRA

Closest papers by meaning

• LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding (CVPR 2024 · similarity 0.92)
• DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.91)
• MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024 · similarity 0.88)
• In-Context Pretraining: Language Modeling Beyond Document Boundaries (ICLR 2024 · similarity 0.87)
• Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation (NeurIPS 2026 · similarity 0.86)
• Graph-based Document Structure Analysis (ICLR 2025 · similarity 0.86)