Research map / Image generation / Text-to-image diffusion
Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FISEdit, a cache-enabled sparse diffusion inference engine that detects text-edit-affected image regions via cross-attention-guided masks and recomputes only those regions, reusing cached features elsewhere, for faster text-to-image editing.
Problem
Interactive text-to-image editing with small prompt modifications requires repeated full diffusion inference, which is slow even on GPUs, despite only a small image region typically changing.
Approach
Automatically derive a fine-grained difference mask from cached vs. edited cross-attention maps (Prompt-to-Prompt style, OTSU thresholding), then run U-Net sparsely on the affected region using adaptive pixel-wise sparse convolution, approximate normalization with cached statistics, and approximate attention, while reusing cached activations for the rest.
Key results
- FISEdit is up to 4.4x faster than baselines on NVIDIA TITAN RTX (8.14 it/s vs 1.83 it/s for SDTP) at 5% edit size, and 3.4x faster on A100 (13.1 vs 3.86 it/s).
- MACs reduced to 386G at 5% edit size versus 1901G for vanilla Stable Diffusion (4.9x reduction).
- Ablation at 768x768 shows the full FISEdit configuration raising throughput from 1.83 to 8.13 it/s on TITAN RTX.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FISEdit for 5% edits instead of SDTP | Stable Diffusion v2 text-to-image editing · LAION-Aesthetics, 5% edit, NVIDIA TITAN RTX | U-Net calls per second: 4.4× | Helped |
| Use FISEdit for 5% edits instead of SDTP | Stable Diffusion v2 text-to-image editing · LAION-Aesthetics, 5% edit, NVIDIA A100 | U-Net calls per second: 3.4× | Helped |
| Add CUDA kernel improvements (ablation) | FISEdit · 512×512 | throughput: 5.42it/s -> 6.80it/s | Helped |
| Add CUDA kernel improvements (ablation) | FISEdit · 768×768 | throughput: 1.83it/s -> 2.73it/s | Helped |
| Add tiled sparse convolution without APSC (ablation) | FISEdit with CUDA kernel improvements · 512×512 | throughput: 6.80it/s -> 6.86it/s | Helped |
| Add tiled sparse convolution without APSC (ablation) | FISEdit with CUDA kernel improvements · 768×768 | throughput: 2.73it/s -> 3.17it/s | Helped |
| Add APSC to tiled sparse convolution (ablation) | FISEdit with CUDA kernel improvements · 512×512 | throughput: 6.86it/s -> 7.2it/s | Helped |
| Add APSC to tiled sparse convolution (ablation) | FISEdit with CUDA kernel improvements · 768×768 | throughput: 3.17it/s -> 3.61it/s | Helped |
| Add approximate attention (ablation) | FISEdit with CUDA kernels and convolution with APSC · 512×512 | throughput: 7.2it/s -> 10.17it/s | Helped |
| Add approximate attention (ablation) | FISEdit with CUDA kernels and convolution with APSC · 768×768 | throughput: 3.61it/s -> 8.13it/s | Helped |
| Apply FISEdit to low-resolution image editing | FISEdit · 256×256 images | editing performance | Hurt |
Surprising findings
- Directly applying tiled sparse convolution (SBNet) fails to give considerable speedup at low resolutions, and the method performs poorly on 256x256 images.
- Caching all per-step activations for 50 DDIM steps can require up to 210 GB, making data transfer a new bottleneck that needed an asynchronous cache pipeline.
Limitations
- Performs poorly on low-resolution images (e.g., 256x256) where edits lack sparsity.
- Caching intermediate activations for all diffusion steps imposes large memory and storage overhead.
- Sparse attention is approximate, with precision errors only controlled by compensation measures.
Experimental setup
Datasets: InstructPix2Pix human-written edit dataset (LAION-Aesthetics based, 454,445 examples)
Benchmarks: InstructPix2Pix edit dataset (LAION-Aesthetics)
Metrics: image-image similarity directional CLIP similarity FID CLIP accuracy MACs it/s
Baselines: SDTP SDIP Prompt-to-Prompt SDEdit InstructPix2Pix DIFFEdit PPAP Pix2Pix-Zero
Builds on: Prompt-to-Prompt SBNet SIGE DIFFEdit
Closest papers by meaning
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.90)
- Inversion-Free Image Editing with Language-Guided Diffusion Models (CVPR 2024 · similarity 0.88)
- Any-Size-Diffusion: Toward Efficient Text-Driven Synthesis for Any-Size HD Images (AAAI 2024 · similarity 0.88)
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.88)
- LEDITS++: Limitless Image Editing using Text-to-Image Models (CVPR 2024 · similarity 0.88)
- Lightning-Fast Image Inversion and Editing for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.