Research map / Large language models / Language, safety and interpretability
Generating Illustrated Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Illustrated Instructions task and StackedDiffusion, which generates step-by-step text with spatially tiled, jointly denoised illustrations from a fine-tuned latent diffusion model plus an LLM for step text.
Problem
Generating customized instructional articles that pair text steps with consistent, goal- and step-faithful illustrations, which naive text-to-image or multimodal LLM approaches fail to produce.
Approach
Spatially tile the latents of all step images into one large image so the diffusion U-Net jointly denoises them for cross-image consistency; concatenate separately encoded goal and step text embeddings with step-positional encodings; initialize from a pretrained T2I model with zero-SNR noise schedule, with no new parameters.
Key results
- StackedDiffusion achieves 74.3% goal faithfulness, 61.5% step faithfulness and 39.5 cross-image consistency vs 22.0/92.9 for frozen T2I in human-style tables; human win rate 78% over finetuned T2I and 66.6% over frozen T2I (as stated in text).
- Human evaluators preferred StackedDiffusion over ground-truth human-made articles 30% of the time.
- Removing the step-positional encoding drops step faithfulness from 61.5% to 49.8%; GILL++ wins only 3.9% against StackedDiffusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate all step images jointly with spatial tiling | StackedDiffusion · Recipes held-out evaluation | cross-image consistency: 51.3 -> 50.7 | Helped |
| use StackedDiffusion instead of T2I (Finetuned) | illustrated instruction generation · Recipes held-out evaluation | step faithfulness accuracy: 52.4 -> 61.5 | Helped |
| use StackedDiffusion instead of T2I (Frozen) | illustrated instruction generation · Recipes held-out evaluation | goal faithfulness accuracy: 92.9 -> 74.3 | Hurt |
| use StackedDiffusion instead of T2I (Frozen) | illustrated instruction generation · Recipes held-out evaluation | FID: 69.3 -> 39.5 | Helped |
| use StackedDiffusion instead of T2I (Frozen) | fully generated outputs · Human evaluation | human evaluation win rate vs StackedDiffusion: 33.8 | Helped |
| remove 0SNR noise schedule adjustment (ablation) | StackedDiffusion · Step faithfulness | step faithfulness accuracy: 61.5% -> 52.8% | Hurt |
| add 0SNR noise schedule adjustment (ablation) | StackedDiffusion · Step faithfulness | step faithfulness accuracy: 52.8% -> 61.5% | Helped |
| remove step-positional encoding (ablation) | StackedDiffusion · Step faithfulness | step faithfulness accuracy: 61.5% -> 49.8% | Hurt |
| add step-positional encoding (ablation) | StackedDiffusion · Step faithfulness | step faithfulness accuracy: 49.8% -> 61.5% | Helped |
| increase training data proportion (ablation) | StackedDiffusion · Illustrated Instructions evaluation | faithfulness and consistency | Helped |
| use 4 steps instead of 6 (ablation) | StackedDiffusion · Human evaluation | human evaluation win rate: 32.1 | Hurt |
| use 8 steps instead of 6 (ablation) | StackedDiffusion · Human evaluation | human evaluation win rate: 46.9 | Hurt |
Surprising findings
- Frozen T2I baseline achieves higher goal faithfulness (92.9%) than the ground truth (81.7%) because it only depicts the finished goal.
- Human evaluators picked StackedDiffusion over human-made ground truth articles 30% of the time.
Limitations
- Gap remains relative to ground-truth human-created illustrated articles.
- Relies on in-house T2I model not publicly available; evaluation restricted to the Recipes category.
- Trained on limited instructional data (WikiHow-based VGSI), max 6 steps.
Experimental setup
Datasets: VGSI (WikiHow articles)
Benchmarks: Illustrated Instructions held-out Recipes split (from VGSI)
Metrics: goal faithfulness accuracy step faithfulness accuracy cross-image consistency (DINO distance) FID human win rate
Baselines: T2I (frozen) T2I (finetuned) GILL GILL++ LLM + CLIP retrieval Goal Retrieval
Builds on: latent diffusion models VGSI classifier-free guidance zero terminal SNR noise schedule
Closest papers by meaning
- DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models (CVPR 2025 · similarity 0.89)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.88)
- AutoPresent: Designing Structured Visuals from Scratch (CVPR 2025 · similarity 0.87)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.87)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.87)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.