Research map / Vision-language and multimodal / Visual question answering
DocVXQA: Context-Aware Visual Explanations for Document Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DocVXQA, a self-explainable DocVQA framework that learns a context-aware visual relevance mask over the document image while answering, using Pix2Struct with ColPali-derived priors.
Problem
Document visual question answering models are black boxes, and existing attention-based explanations are noisy and highlight answer occurrences without the context justifying the answer.
Approach
Learn a mask over the document image (trained via information-bottleneck-inspired sufficiency, minimality via L1 and continuity losses, and an MSE alignment to ColPali heatmap priors) and feed the masked image back into the same fine-tuned Pix2Struct to answer, making explanations self-explainable and context-aware.
Key results
- On DocVQA at threshold 0.7 with postprocessing, the method reaches 0.38 accuracy and 0.54 ANLS at 0.23 pixel ratio, beating ColPali+Pix2Struct (0.38/0.50 at 0.23).
- On PFL-DocVQA, the method reaches 0.43 accuracy and 0.66 ANLS at 0.22 pixel ratio versus 0.28/0.39 at 0.18 for ColPali+Pix2Struct.
- In a 42-participant human study, the method scored 4.49 on context-awareness and 3.56 on clarity, above ColPali+Pix2Struct (3.97, 3.02); in pairwise preference it was chosen in 64.7% of trials.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add minimality loss to sufficiency loss (ablation) | DocVXQA · DocVQA | Accuracy: 0.51 -> 0.19 | Hurt |
| Add minimality loss to sufficiency loss (ablation) | DocVXQA · DocVQA | Pixel Ratio: 1 -> 0.02 | Helped |
| Add token interactions loss to sufficiency and minimality losses (ablation) | DocVXQA · DocVQA | Accuracy: 0.19 -> 0.38 | Helped |
| Add token interactions loss to sufficiency and minimality losses (ablation) | DocVXQA · DocVQA | ANLS: 0.36 -> 0.54 | Helped |
| Increase the number of retained boxes from k=1 to k=100 (ablation) | DocVXQA postprocessing · DocVQA | Accuracy: 0.36 -> 0.45 | Helped |
| Increase the number of retained boxes from k=1 to k=100 (ablation) | DocVXQA postprocessing · DocVQA | Pixel Ratio: 0.21 -> 0.40 | Hurt |
| Use learned context-aware explanations instead of ColPali+Pix2Struct explanations | DocVXQA · Explanation human rating | Context: 3.97 ± 0.25 -> 4.49 ± 0.26 | Helped |
| Use learned context-aware explanations instead of ColPali+Pix2Struct explanations | DocVXQA · Explanation human rating | Clarity: 3.02 ± 0.42 -> 3.56 ± 0.41 | Helped |
Surprising findings
- Minimality loss alone (S+M) collapsed accuracy to 0.19 and ANLS to 0.36 at a 0.02 pixel ratio, showing masks overfit to answer tokens without token-interaction priors.
- The mutual-information estimator via matrix-based Renyi entropy did not yield expected results, so the authors switched to an L1 plus anisotropic total variation continuity loss.
Limitations
- Accuracy and ANLS drop relative to the unmasked Pix2Struct model (0.51 vs 0.56 accuracy on DocVQA for unmasked comparison), a cost of masking.
- Mutual-information-based minimality was unstable in experiments, requiring a heuristic L1/TV alternative.
- Explanation quality depends on the ColPali prior, whose attention maps are noted to often fail to match human intuition.
Experimental setup
Datasets: DocVQA PFL-DocVQA InfographicVQA
Benchmarks: DocVQA PFL-DocVQA
Metrics: accuracy ANLS pixel ratio human context-awareness rating human clarity rating human preference rate
Baselines: Raw Attention Attention Rollout Grad-CAM ColPali+Pix2Struct Pix2Struct unmasked
Builds on: Pix2Struct ColPali Donut Attention Rollout Grad-CAM
Closest papers by meaning
- DocPrune: Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruning (CVPR 2026 · similarity 0.89)
- WebQA: Multihop and Multimodal QA (CVPR 2022 · similarity 0.88)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.88)
- Separation of Powers: On Segregating Knowledge from Observation in LLM-enabled Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.88)
- Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering (NeurIPS 2025 · similarity 0.88)
- Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understanding (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.