Research map / Image generation / Fast diffusion sampling

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision

ICML 2025 · Image generation › Fast diffusion sampling · approach: Diffusion models as perception tools

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TimeStep Master (TSM), which trains separate LoRA experts per diffusion-timestep interval and then assembles them into an asymmetric mixture (a core expert without gating plus gated context experts) to improve LoRA-tuned diffusion models on domain adaptation, post-pretraining, and distillation.

Problem

Vanilla LoRA applies one set of low-rank weights across all diffusion timesteps, which limits its ability to model different noise levels, degrading generation quality in downstream adaptation, post-pretraining, and distillation.

Approach

Fine-tune a separate LoRA per timestep interval (fostering stage), then combine experts at multiple interval scales with a finest-scale core expert ungated and timestep-dependent gating over coarser context experts (assembling stage), training only the router in stage two.

Key results

• On T2I-CompBench domain adaptation, SD3 + TSM reaches Color 83.45 and Spatial 34.50 versus 82.41 and 31.87 for SD3 + vanilla LoRA.
• In model distillation on DMD2 with SD1.5, DMD2 + TSM achieves FID 9.90 versus 14.58 for DMD2 + LoRA at 3.7 vs 3.6 A100 days.
• In video post-pretraining on VideoCrafter2, TSM raises Color from 41.30 to 60.87 and Count from 27.89 to 60.38 relative to vanilla LoRA.

Measured effects reported in the paper

Surprising findings

• The number of context LoRA experts and the 1-stage granularity have little effect on 2-stage performance, so the simplest setting with one context expert is used.
• Symmetric mixtures without a core expert often perform worse than the asymmetric design.

Limitations

• Gains on some categories (e.g., complex domain in domain adaptation for some models, Complex score on VC2 post-pretraining) are small or negative in ablations.
• Distillation result is not the lowest FID among compared methods due to limited training resources, per the authors.
• Ablations are run on a limited set of models and tasks, with the video setting using a 70k subset.

Experimental setup

Datasets: T2I-CompBench SAM-LLaVA-Captions10M OpenVid-1M COCO2014 LAION-Aesthetic EvalCrafter

Benchmarks: T2I-CompBench EvalCrafter COCO2014

Metrics: T2I-CompBench attribute scores (color, shape, texture, spati Inception Score Action Amplitude BLIP-BLEU Color Count FID Patch-FID CLIP score

Baselines: Vanilla LoRA MoE LoRA LCM-LoRA DMD2 DMD2 + LoRA SD1.5 PixArt-α-ft SD3

Builds on: LoRA DMD2 mixture of experts PEFT

Closest papers by meaning

• Adaptive Non-Uniform Timestep Sampling for Accelerating Diffusion Model Training (CVPR 2025 · similarity 0.90)
• LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation (ICLR 2025 · similarity 0.89)
• LoRAtorio: An intrinsic approach to LoRA Skill Composition (NeurIPS 2026 · similarity 0.88)
• TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models (AAAI 2025 · similarity 0.88)
• Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization (ICLR 2025 · similarity 0.88)
• Improving Training Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architecture (CVPR 2024 · similarity 0.88)