Research map / Vision-language and multimodal / Human and embodied modeling
CollageNoter: Real-Time and Adaptive Collage Layout Design for Screenshot-Based E-Note-Taking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CollageNoter, a two-stage readability-then-cognitive pipeline for screenshot collage E-Note layouts, using a GAN-style cascade transformer (CollageFormer) trained on a newly built device-adaptive CollageLayout dataset.
Problem
Collage layout design for screenshot-based e-note-taking: semantic relevance-first tree methods cause visual distortion and conflict with human reading order, and existing graphic layout datasets are unsuitable for collage layouts across device sizes.
Approach
A readability-first transformer layout generator (cascade position-then-size prediction with an adversarial discriminator) that preserves screenshot aspect ratios, followed by a cognitive-driven adjustment using a tree-based F-pattern reading-order reassignment and alignment-aware grouping, plus a size-aware dataset builder for arbitrary device resolutions.
Key results
- On HD, CollageFormer reaches Miou 0.873 vs 0.831 for BLT (best baseline) with Over 0.007 and Empty 0.026.
- Running time under 2 seconds at 1920x1080 canvas, versus 61-488 seconds for Softcollage and 8-60 seconds for Semi depending on input count.
- In a 12-participant user study, 83.3% preferred CollageNoter layouts as most visually appealing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CollageFormer instead of BLT | collage layout generation · HD | Miou: 0.831 -> 0.873 | Helped |
| use CollageFormer instead of BLT | collage layout generation · UHD | Miou: 0.686 -> 0.723 | Helped |
| use CollageFormer instead of BLT | collage layout generation · UR | Miou: 0.812 -> 0.849 | Helped |
| add the cascade generator to the layout model (ablation) | CollageFormer · HD | Miou: 0.847 -> 0.873 | Helped |
| add the cascade generator to the layout model (ablation) | CollageFormer · UHD | Miou: 0.706 -> 0.723 | Helped |
| add the discriminator to reconstruction-loss training (ablation) | CollageFormer · HD | Miou: 0.839 -> 0.873 | Helped |
| add the discriminator to reconstruction-loss training (ablation) | CollageFormer · UHD | Miou: 0.689 -> 0.723 | Helped |
| use CollageFormer instead of BLT | collage layout generation · HD | Over: 0.011 -> 0.007 | Helped |
| use CollageFormer instead of BLT | collage layout generation · UHD | Over: 0.037 -> 0.031 | Helped |
| use CollageFormer instead of BLT | collage layout generation · UR | Over: 0.019 -> 0.012 | Helped |
| use CollageNoter rather than the baseline collage layouts | E-Note layout selection · user study | participant preference: 83.3% | Helped |
Surprising findings
- The cascade position-then-size design alone (Ours-cascade) performed worse than the variant without the discriminator on some metrics, which the authors note as a point about the cascade framework's importance.
Limitations
- Relies on hand-constructed reading-order heuristics (F-pattern) rather than learned cognitive preferences.
- Only evaluated on a small number of input screenshots in the user study (6 and 4 groups).
Experimental setup
Datasets: CollageLayout Magazine Rico PublayNet PKU
Benchmarks: CollageLayout (HD, UHD, UR)
Metrics: Miou Over Align Empty running time user preference rate answer search time
Baselines: Trans BLT Former++ Softcollage Semi-automatic layout adaptation
Builds on: Former++ BLT Gupta et al. transformer layout generation SoftCollage
Closest papers by meaning
- Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement (CVPR 2026 · similarity 0.81)
- PosterO: Structuring Layout Trees to Enable Language Models in Generalized Content-Aware Layout Generation (CVPR 2025 · similarity 0.81)
- Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.81)
- SpotActor: Training-Free Layout-Controlled Consistent Image Generation (AAAI 2025 · similarity 0.81)
- Paper2Poster: Towards Multimodal Poster Automation from Scientific Papers (NeurIPS 2025 · similarity 0.80)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.