Research map / Image generation / Text-to-image diffusion
Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LIGER, a training-free framework that generates long-horizon visual instructions step by step using historical prompts, visual memory, and MLLM-guided tool-based self-reflection with inversion-based memory calibration, and introduces a 569-task benchmark with human-written expressions.
Problem
Generating a coherent and logically accurate series of images as visual instructions for long-horizon tasks, where existing text-to-image generation produces inconsistent images, misses objects, and gets object attributes wrong.
Approach
Combine a frozen SDXL with LLM-generated historical prompts and shared visual memory for step-to-step continuity, then let GPT-4o detect errors (over-consistency, redundant objects, identity inconsistency, wrong attributes) and call editing tools (LISA, DragonDiffusion, SD inpainting, LAMA) to fix them, and calibrate visual memory via DDIM inversion of the revised image.
Key results
- LIGER achieves CLIP-Score 2.7555, DINO-Score 0.6338 (lower is better), and BERT-Score 0.8743, outperforming T2I-Bridge and SLDM.
- In GPT-4o evaluation, LIGER scores 4.1141 semantic, 3.0595 logic, and 3.0536 illustrative, versus SLDM's 2.7286 logic.
- In a 20-participant user study, LIGER is preferred in 64.3% semantic, 60.7% logic, and 68.3% illustrative comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add historical prompts and visual memory to SDXL (ablation) | SDXL · Long-horizon task benchmark | CLIP-Score: 2.5837 -> 2.7168 | Helped |
| Add historical prompts and visual memory to SDXL (ablation) | SDXL · Long-horizon task benchmark | DINO-Score: 0.8516 -> 0.7459 | Helped |
| Add historical prompts and visual memory to SDXL (ablation) | SDXL · Long-horizon task benchmark | BERT-Score: 0.8699 -> 0.8721 | Helped |
| Add visual memory alone (ablation) | SDXL · Long-horizon task benchmark | CLIP-Score: 2.5837 -> 2.6251 | Helped |
| Add visual memory alone (ablation) | SDXL · Long-horizon task benchmark | DINO-Score: 0.8516 -> 0.8239 | Helped |
| Add historical prompts alone (ablation) | SDXL · Long-horizon task benchmark | CLIP-Score: 2.5837 -> 2.6842 | Helped |
| Add historical prompts alone (ablation) | SDXL · Long-horizon task benchmark | DINO-Score: 0.8516 -> 0.8224 | Helped |
| Add self-reflection to SDXL with visual memory and historical prompts (ablation) | SDXL+V+H · Long-horizon task benchmark | CLIP-Score: +0.04 | Helped |
| Add self-reflection to SDXL with visual memory and historical prompts (ablation) | SDXL+V+H · Long-horizon task benchmark | DINO-Score: -0.112 | Helped |
| Add self-reflection to SDXL with visual memory and historical prompts (ablation) | SDXL+V+H · Long-horizon task benchmark | BERT-Score: +0.002 | Helped |
| Add inversion-guided visual memory calibration after image editing (ablation) | LIGER · Subsequent step after seasoning the steak | Raw-state correctness | Helped |
| Use LIGER instead of the strongest baseline | Long-horizon task benchmark · Long-horizon task benchmark | CLIP-Score: 2.5054 -> 2.7555 | Helped |
Limitations
- Relies on proprietary GPT-4o for error detection, historical prompts, and judging.
- Each step's self-reflection is applied once, so errors not caught in the single pass persist.
- Evaluated only on recipe-domain tasks.
Experimental setup
Datasets: Howto100M YouCook2 RecipeQA
Benchmarks: LIGER long-horizon benchmark (569 tasks)
Metrics: CLIP-Score DINO-Score BERT-Score GPT-4o rating (semantic, logic, illustrative) user study preference
Baselines: Frozen SDXL T2I-Bridge SLDM
Builds on: StoryDiffusion SDXL LISA DragonDiffusion
Closest papers by meaning
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.88)
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning (ICLR 2024 · similarity 0.87)
- Text-Vision Co-Instructed Image Editing (NeurIPS 2026 · similarity 0.87)
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.87)
- Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels (CVPR 2025 · similarity 0.86)
- Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.