Research map / Video / Video-language understanding
Track the Answer: Extending TextVQA from Image to Video with Spatio-Temporal Clues
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends generative T5-based image TextVQA to video by recovering spatio-temporal relations among OCR and object entities (temporal convolution, OCR-enhanced relative spatial bias) and aggregating question-aware scene-text clues, reaching SOTA on M4-ViteVQA and RoadTextVQA.
Problem
Video text-based VQA requires reasoning over scene text across frames, but existing generative approaches flatten visual entities into a 1-D sequence, disrupting spatio-temporal relations and being distracted by redundant frames.
Approach
Adds a temporal convolution module between T5 attention layers to link same-instance entities across frames, a 2-D relative spatial attention bias using word/line/paragraph OCR boxes, and a question-driven scene-text-aware clue aggregation (CLIP global context + question self-attention + cross-attention to OCR) that is fed to the decoder.
Key results
- TEA-Large reaches 34.78% accuracy and 43.71% ANLS on M4-ViteVQA Task1Split1 test, versus 22.17% for T5-ViteVQA.
- TEA-Large achieves 48.14% accuracy on the RoadTextVQA validation set, versus 30.82% for fine-tuned Video-LLaVA.
- In ablation, the OCR-enhanced relative spatial module adds 5.12% accuracy over the baseline, and combined spatio-temporal recovery adds 6.90%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace T5-ViteVQA with TEA-Large | Video TextVQA · M4-ViteVQA Task1Split1 test | accuracy: 34.78 vs. 22.17 | Helped |
| add temporal convolution (ablation) | TEA-Baseline · M4-ViteVQA Task1Split1 validation | accuracy: 26.79 -> 28.56 | Helped |
| add OCR-enhanced relative spatial module (ablation) | TEA-Baseline · M4-ViteVQA Task1Split1 validation | accuracy: 26.79 -> 31.91 | Helped |
| add temporal convolution and OCR-enhanced relative spatial module (ablation) | TEA-Baseline · M4-ViteVQA Task1Split1 validation | accuracy: 26.79 -> 33.69 | Helped |
| add scene text-aware clue aggregation (ablation) | TEA-Baseline · M4-ViteVQA Task1Split1 validation | accuracy: 26.79 -> 29.22 | Helped |
| add scene text-aware clue aggregation to temporal and spatial features (ablation) | Temporal + Spatial TEA · M4-ViteVQA Task1Split1 validation | accuracy: 33.69 -> 34.45 | Helped |
| replace multi-granularity spatial relationships with word-level relationships (ablation) | OCR-enhanced relative spatial module · M4-ViteVQA Task1Split1 validation | accuracy: 31.91 -> 31.06 | Hurt |
| add question features to global visual semantics (ablation) | Scene text-aware clue aggregation · M4-ViteVQA Task1Split1 validation | accuracy: 26.79 -> 27.40 | Helped |
| add OCR and object features to question-aware global semantics (ablation) | Scene text-aware clue aggregation · M4-ViteVQA Task1Split1 validation | accuracy: 27.40 -> 28.46 | Helped |
| exclude object features from question- and OCR-aware clue aggregation (ablation) | Scene text-aware clue aggregation · M4-ViteVQA Task1Split1 validation | accuracy: 28.46 -> 29.22 | Helped |
Surprising findings
- Video-LLMs fine-tuned with LoRA underperform dedicated Video TextVQA methods, which the authors attribute to their inability to read small scene text.
- Adding object features to the clue aggregation module did not help; scene text plus question alone gave the best clue features.
Limitations
- Relies on external OCR and object detection systems whose errors propagate to the model.
- Video-LLM baselines were only fine-tuned with LoRA, which may understate their potential.
Experimental setup
Datasets: M4-ViteVQA RoadTextVQA
Benchmarks: M4-ViteVQA RoadTextVQA
Metrics: accuracy ANLS
Baselines: M4C JuskAsk All-in-one-B GIT SINGULARITY Video-LLaVA VideoLLaMA2 T5-ViteVQA
Builds on: T5 T5-ViteVQA LaTr LoRA
Closest papers by meaning
- VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · similarity 0.90)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.90)
- Separation of Powers: On Segregating Knowledge from Observation in LLM-enabled Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.88)
- Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models (NeurIPS 2026 · similarity 0.88)
- Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.88)
- Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.