Research map / Image generation / Text-to-image diffusion
Deep Submodular Optimization and LLM for Multimodal Content Extraction and Automatic Poster Generation from Long Document
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PostDoc, a pipeline that selects multimodal content from a long document using a trainable deep submodular function (coverage, diversity, text-image alignment), paraphrases it with GPT-3.5-turbo, and generates poster templates with heuristic and model-based design components.
Problem
Automatically transforming a long multimodal (text and image) document into a one-page poster, which requires content selection with coverage, diversity and multimodal alignment plus template/layout generation.
Approach
A multimodal deep submodular objective over BLIP embeddings with coverage, diversity, and image-text alignment terms, whose non-negative dimension weights are trained from ground-truth summaries via hinge loss and alternating greedy selection and projected SGD; paired with a single GPT-3.5-turbo paraphrase call and rule/model-based template design.
Key results
- On NJU-Fudan, PostDoc reaches ROUGE-L 0.50 vs 0.13 for GPT-3.5+BLIP and image precision 0.53 vs 0.39 for baselines, while running about 10x faster than GPT-3.5+BLIP (4.25 vs 47.37 s).
- On the Let Me Choose font dataset, PostDoc's Top-1 F1 is 0.4301 vs 0.2697 for the BERT baseline.
- In a small human study (3 reviewers, 5 papers), PostDoc's template rating was 3.46 vs 2.33 for GPT-3.5+BLIP and run-time rating 3.60 vs 1.86.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the deep submodular function with a feature-based submodular function (ablation) | PostDoc · MSMO test | ROUGE-L: 0.57 ± 0.13 -> 0.68 ± 0.14 | Helped |
| replace the deep submodular function with a feature-based submodular function (ablation) | PostDoc · NJU-Fudan | ROUGE-L: 0.43 ± 0.10 -> 0.50 ± 0.04 | Helped |
| add the coverage term (ablation) | PostDoc · MSMO test | ROUGE-L: 0.51 ± 0.08 -> 0.68 ± 0.14 | Helped |
| add the coverage term (ablation) | PostDoc · NJU-Fudan | ROUGE-L: 0.48 ± 0.05 -> 0.50 ± 0.04 | Helped |
| add the diversity term (ablation) | PostDoc · MSMO test | ROUGE-L: 0.50 ± 0.13 -> 0.68 ± 0.14 | Helped |
| add the diversity term (ablation) | PostDoc · NJU-Fudan | ROUGE-L: 0.48 ± 0.05 -> 0.50 ± 0.04 | Helped |
| add the multimodal alignment term (ablation) | PostDoc · MSMO test | ROUGE-L: 0.56 ± 0.12 -> 0.68 ± 0.14 | Helped |
| add the multimodal alignment term (ablation) | PostDoc · NJU-Fudan | ROUGE-L: 0.45 ± 0.04 -> 0.50 ± 0.04 | Helped |
| add the coverage term (ablation) | PostDoc · MSMO test | Coverage: 0.24 ± 0.06 -> 0.30 ± 0.03 | Helped |
| add the coverage term (ablation) | PostDoc · NJU-Fudan | Coverage: 0.43 ± 0.04 -> 0.42 ± 0.03 | Mixed |
| add the multimodal alignment term (ablation) | PostDoc · NJU-Fudan | IP: 0.34 ± 0.19 -> 0.53 ± 0.32 | Helped |
| use PostDoc instead of BRIO + BLIP | multimodal summarization · MSMO test | ROUGE-L: 0.36 ± 0.11 -> 0.68 ± 0.14 | Helped |
Surprising findings
- On MSMO, PostDoc's ROUGE-L (0.68) is higher than the GPT-3.5 baseline's, but its image precision is on par with or slightly below the baselines, so gains were not uniform across metrics.
- Ablations removing individual submodular terms showed no consistent pattern on coverage, diversity, and image precision.
Limitations
- Performance is limited for non-natural images such as flow charts and neural diagrams, and structured elements like tables.
- Human evaluation is small-scale (3 reviewers, 5 papers).
- Could not replicate or compare end-to-end against existing document-to-poster systems due to lack of code.
Experimental setup
Datasets: MSMO NJU-Fudan Let Me Choose
Benchmarks: MSMO NJU-Fudan Let Me Choose
Metrics: ROUGE-L coverage diversity image precision inference time top-1 F1 top-3 F1 NGOMetric human ratings (1-5)
Baselines: MemSum + BLIP BRIO + BLIP GPT-3.5 + BLIP PostDoc w/o dsf Let Me Choose BERT Model LayoutDM
Builds on: deep submodular functions BLIP MemSum BRIO LayoutDM
Closest papers by meaning
- SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation (CVPR 2025 · similarity 0.84)
- LLMs can see and hear without any training (ICML 2025 · similarity 0.84)
- DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.83)
- BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks (ICLR 2025 · similarity 0.83)
- SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.83)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.