Research map / Label-efficient and robust learning / Recognition and long-tail
KIND: Knowledge Integration and Diversion for Training Decomposable Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KIND, a pre-training method that uses an SVD-structured DiT whose components are split into class-agnostic learngenes and class-specific tailors via a class gate, yielding decomposable diffusion transformers for variable-size deployment and domain-shift transfer.
Problem
Pre-trained models are fixed in size and prone to negative transfer under large domain shifts; deploying them across resource budgets and tasks is inflexible.
Approach
Reparameterize each DiT weight matrix via SVD-style components (column vector, singular value, row vector) and, during pre-training, route gradients by class so that shared components become learngenes and per-class components become tailors, producing a model that can be decomposed and recombined.
Key results
- On training classes, KIND-DiT-L reaches FID 9.33 with 245.9M params versus 9.68 for traditional pre-training at 457.0M, with zero additional training steps.
- On 18 novel-class downstream tasks with DiT-L/2, KIND reduces FID to 12.87 versus 22.55 for LoRA and 19.41 for PiSSA, using 45.4M trainable parameters.
- On large-domain-shift tasks, KIND achieves the lowest FDD across CelebA-HQ, Hubble, MRI and Pokemon for both DiT-B and DiT-L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KIND instead of traditional pre-training for decomposable models | DiT-L · ImageNet-1K training classes | FID: 9.68 -> 9.33 | Helped |
| Use KIND instead of traditional pre-training for decomposable models | DiT-B · ImageNet-1K training classes | FID: 25.14 -> 21.14 | Helped |
| Transfer learngenes with randomly initialized tailors | DiT-B/2 · ImageNet-1K novel classes | FID: 33.16 -> 20.94 | Helped |
| Transfer learngenes with randomly initialized tailors | DiT-L/2 · ImageNet-1K novel classes | FID: 14.51 -> 12.87 | Helped |
| Transfer learngenes with randomly initialized tailors instead of PEFT | DiT-B and DiT-L · CelebA-HQ, Hubble, MRI, and Pokemon | FDD: CelebA-HQ: 0.282 -> 0.201 (DiT-B), 0.195 -> 0.152 (DiT-L); Hubble: 0.211 -> 0.124 (DiT-B), 0.142 -> 0.109 (DiT-L); MRI: | Helped |
| Add KIND learngenes in place of a randomly selected SVD backbone (ablation) | DiT-B/2 · ImageNet-1K novel classes | FID: 60.28 -> 49.54 | Helped |
| Add KIND learngenes in place of a randomly selected SVD backbone (ablation) | DiT-L/2 · ImageNet-1K novel classes | FID: 42.04 -> 33.53 | Helped |
| Replace LoRA with tailors (ablation) | KIND learngene backbone on DiT-B/2 · ImageNet-1K novel classes | FID: 49.54 -> 21.60 | Helped |
| Replace LoRA with tailors (ablation) | KIND learngene backbone on DiT-L/2 · ImageNet-1K novel classes | FID: 33.53 -> 13.03 | Helped |
| Add the class gate to the learngene-and-tailor method (ablation) | DiT-B/2 · ImageNet-1K novel classes | FID: 21.60 -> 20.94 | Helped |
| Add the class gate to the learngene-and-tailor method (ablation) | DiT-L/2 · ImageNet-1K novel classes | FID: 13.03 -> 12.87 | Helped |
Surprising findings
- Learngene-only backbones show higher entropy and lower variance and kurtosis than the full pre-trained model, which the authors read as more class-agnostic knowledge.
Limitations
- Evaluated only on class-conditional image generation with DiT backbones at 256x256 resolution.
- Requires training with class labels, so the class gate only applies to labeled, class-partitioned data.
- Some PEFT baselines and downstream results have high variance across tasks, such as DoRA on Hubble.
Experimental setup
Datasets: ImageNet-1K CelebA-HQ Hubble MRI Pokemon
Benchmarks: ImageNet-1K class-conditional generation (training classes) 18 downstream novel-class tasks CelebA-HQ Hubble MRI Pokemon
Metrics: FID sFID IS Precision Recall FDD
Baselines: Traditional pre-training Heur-LG Auto-LG Laptop-diff SVDiff OFT LoRA PiSSA LoHa DoRA Full FT
Builds on: Learngene Diffusion Transformers (DiT) FSGAN LoRA PiSSA
Closest papers by meaning
- MergeNet: Knowledge Migration Across Heterogeneous Models, Tasks, and Modalities (AAAI 2025 · similarity 0.84)
- SEMU: Singular Value Decomposition for Efficient Machine Unlearning (ICML 2025 · similarity 0.83)
- KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models (ICLR 2025 · similarity 0.83)
- Knowledge Composition using Task Vectors with Learned Anisotropic Scaling (NeurIPS 2024 · similarity 0.83)
- CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning (NeurIPS 2024 · similarity 0.83)
- Amalgamating Multi-Task Models with Heterogeneous Architectures (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.