Research map / Architectures and efficiency / Attention and transformers
CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Linearizes pre-trained FLUX-1.dev by replacing full attention with a circular local-window attention (CLEAR) and distilling only attention-layer weights on 10K self-generated samples.
Problem
Quadratic attention in pre-trained Diffusion Transformers makes high-resolution image generation very slow; it is unclear which efficient attention methods can be applied to pre-trained DiTs without quality loss.
Approach
Identifies four factors (locality, formulation consistency, high-rank attention maps, feature integrity) needed to linearize pre-trained DiTs, and proposes CLEAR: each image query attends only to text tokens and key-values inside a circular local window of radius r, distilled from the original model's predictions and layer attention outputs.
Key results
- CLEAR (r=32) with distillation reaches FID 8.85 against original FLUX-1.dev outputs on COCO 1024x1024 (vs 18.90 for Swin Transformer), with CLIP-I 92.18.
- At 8192x8192, CLEAR reduces attention computation by 99.5% and accelerates generation 6.3x over FLUX-1.dev (per abstract; running time 293.5s vs 1842.5s per 50 steps for r=8).
- Distilled CLEAR layers transfer zero-shot to FLUX-1.schnell (FID 12.43 vs original at r=32) and to a pre-trained ControlNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLEAR instead of full attention | FLUX-1.dev · COCO2014 validation, 1024×1024 | FID against original: — -> 8.85 | Helped |
| Add knowledge distillation (ablation) | CLEAR (r=8) on FLUX-1.dev · COCO2014 validation, 1024×1024 | FID against original: 15.53 -> 13.07 | Helped |
| Add knowledge distillation (ablation) | CLEAR (r=16) on FLUX-1.dev · COCO2014 validation, 1024×1024 | FID against original: 14.27 -> 13.72 | Helped |
| Add knowledge distillation (ablation) | CLEAR (r=32) on FLUX-1.dev · COCO2014 validation, 1024×1024 | FID against original: 11.07 -> 8.85 | Helped |
| Increase CLEAR window radius from r=8 to r=32 (ablation) | CLEAR on FLUX-1.dev · COCO2014 validation, 1024×1024 | FID against original: 13.07 -> 8.85 | Helped |
| Replace full attention with Sigmoid Attention | FLUX-1.dev · COCO2014 validation, 1024×1024 | FID against original: — -> 447.80 | Hurt |
| Train CLEAR using self-generated samples instead of real LAION images (ablation) | FLUX-1.dev · CLEAR fine-tuning | training loss | Helped |
| Approximate text-token attention with patch-wise averaging (ablation) | CLEAR multi-GPU inference · COCO2014 validation, 1024×1024 | CLIP-T: 30.65 -> 31.21 | Mixed |
| Increase the number of inference patches from 2 to 8 (ablation) | CLEAR (r=8) multi-GPU inference · COCO2014 validation, 1024×1024 | CLIP-T: 31.21 -> 31.22 | No clear effect |
| Increase CLEAR window radius from r=8 to r=32 (ablation) | CLEAR on FLUX-1.dev · COCO2014 validation, 2048×2048 | PSNR: 27.57 -> 28.95 | Helped |
| Apply FLUX-1.dev-trained CLEAR layers to FLUX-1.schnell without adaptation | FLUX-1.schnell · COCO2014 validation, 1024×1024 | FID against original: 12.43 | Mixed |
| Apply CLEAR to pretrained ControlNet | FLUX-1.dev ControlNet with grayscale conditions · COCO2014 validation | FID against original: 11.57 | Mixed |
Surprising findings
- Formulation-variation methods such as linear attention and sigmoid attention perform poorly on pre-trained DiTs, unlike on UNets or DiTs trained from scratch.
- Training on 10K self-generated FLUX samples yields much better results than training on a 200K real LAION subset.
Limitations
- Practical speedup lags theoretical FLOPS reduction; sparse attention is hard to optimize versus FlashAttention.
- CLEAR can be slower than the original DiT below roughly 1024x1024 resolution.
- Lacking global token interactions may hurt global structural properties such as symmetry.
Experimental setup
Datasets: FLUX.1-dev self-generated 10K samples LAION5B subset (200K, aesthetic>7) for comparison COCO2014 validation
Benchmarks: COCO2014 validation
Metrics: FID LPIPS CLIP-I DINO CLIP-T IS GFLOPS PSNR SSIM RMSE
Baselines: Sigmoid Attention Linear Attention PixArt-Sigma Agent Attention Strided Attention Swin Transformer
Builds on: Neighborhood Attention FLUX.1-dev FlexAttention DistriFusion LinFusion
Closest papers by meaning
- DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling (NeurIPS 2025 · similarity 0.92)
- DiTFastAttn: Attention Compression for Diffusion Transformer Models (NeurIPS 2024 · similarity 0.90)
- DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention (CVPR 2025 · similarity 0.90)
- Parameterized Stripe Attention for Efficient Video Generation (NeurIPS 2026 · similarity 0.89)
- Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.89)
- SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.