Research map / Label-efficient and robust learning / Recognition and long-tail
ETDPC: A Multimodality Framework for Classifying Pages in Electronic Theses and Dissertations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ETDPC, a two-stream ResNet50v2 + BERT-with-Talking-Heads cross-attention multimodal classifier with paraphrase/image augmentation and a two-level hierarchy to classify ETD pages into 13 categories.
Problem
Classifying pages of electronic theses and dissertations (ETDs) into 13 structural categories, where existing document-layout models perform poorly and minority categories have very few labeled samples.
Approach
A two-stream multimodal model (ResNet50v2 visual encoder and BERT with Talking-Heads text encoder over OCR text) fused by early fusion plus a cross-attention layer, combined with augmentation of minority pages (PEGASUS paraphrasing, text-to-image rendering, ScanBank-style noise) and a two-level chapter/non-chapter classifier.
Key results
- ETDPC reached 0.85 accuracy and 0.83 macro F1 on the ETD500 test set, versus 0.60 accuracy for VGG16 and 0.28 for DocFormer.
- With the two-level classifier plus augmentation, F1 reached 0.84-0.96 on 9 of 13 categories.
- One-level classifier achieved 0.92 F1 on Chapters but near-zero F1 on minority categories such as ChapterAbstract and Abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a two-level classifier instead of a one-level classifier | ETDPC · ETD500 test set | macro F1: 0.53 -> 0.68 | Helped |
| add 5,984 augmented samples to the two-level classifier training data | ETDPC · ETD500 test set | macro F1: 0.68 -> 0.83 | Helped |
| replace BERT-large with BERT with Talking-Heads Attention (ablation) | ETDPC multimodal model · ETD500 | F1: 0.02 to 0.43 | Helped |
| use both visual and textual modalities with cross-attention instead of either modality alone (ablation) | ETDPC · ETD500 test set | F1: 0.10 -- 0.17 when compared to only visual modality; 0.02 -- 0.17 when compared to only textual modality | Helped |
| increase training samples for minority categories | ETDPC Case c · ETD500, ListofTables | F1: ∼0.05; almost 0.10 | Mixed |
| increase training samples for minority categories | ETDPC Case c · ETD500, ListofFigures | F1: about 0.05 | Mixed |
| increase training samples for minority categories | ETDPC Case c · ETD500, Abstract | F1: ∼0.02 | Mixed |
| increase training samples | ETDPC Case c · ETD500, Dedication | F1 | No clear effect |
| increase training samples | ETDPC Case c · ETD500, TitlePage | F1: ∼0.02 | Mixed |
Surprising findings
- Fine-tuned LayoutLMv2 and DocFormer performed poorly on ETD pages (below 30% accuracy), worse than the simple visual VGG16 baseline.
- ListofTables and ListofFigures were frequently confused with each other because the pages look visually similar apart from headings.
Limitations
- Minority categories such as ListofTables and ListofFigures still have low F1 due to visual similarity and limited data.
- Evaluated only on scanned ETDs from 1945-1990; born-digital ETDs not tested.
- Hyperparameters were heuristically chosen.
Experimental setup
Datasets: ETD500 RVL-CDIP ETD-OD
Benchmarks: ETD500
Metrics: precision recall F1 accuracy macro F1
Baselines: LayoutLMv2 DocFormer VGG16
Builds on: ResNet BERT talking-heads attention PEGASUS ScanBank
Closest papers by meaning
- DocSAM: Unified Document Image Segmentation via Query Decomposition and Heterogeneous Mixed Learning (CVPR 2025 · similarity 0.81)
- OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations (CVPR 2025 · similarity 0.81)
- Graph-based Document Structure Analysis (ICLR 2025 · similarity 0.81)
- AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generation (NeurIPS 2026 · similarity 0.80)
- OAK: Enriching Document Representations using Auxiliary Knowledge for Extreme Classification (ICML 2024 · similarity 0.80)
- Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.