Research map / Image generation / Fast diffusion sampling
Diffusion-based Neural Network Weights Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D2NWG, a latent diffusion model that generates dataset- or task-conditioned neural network weights (including LoRA and LLM layers) from zoos of pretrained weights, without fine-tuning.
Problem
Transfer learning is limited by costly model selection and storage of many pretrained models; existing weight generators are small-scale, focus on in-distribution sampling, or need task-specific model collections or original training data.
Approach
Learn the distribution of pretrained weights with a VAE over weight vectors (model-wise, layer-wise, or chunk-wise), then train a latent diffusion UNet conditioned on Set Transformer dataset embeddings (aligned to weight embeddings via a CLIP-style loss) to sample weights for new datasets or task descriptions; extend to LLM layers chosen by the spectrum/Marchenko-Pastur method.
Key results
- On mini-ImageNet 5-way 1-shot with ResNet12, D2NWG reaches 69.55% vs 64.99% for MetaDiff (Table 2).
- Zero-shot classifier head generation on CIFAR-10 with ResNet18 reaches 60.42% with the CLIP-based variant vs 10.88% for the pretrained ResNet18 baseline.
- On Open LLM Leaderboard v2, the D2NWG variant of Llama-3.1-8B-Instruct improves average score from 27.91 to 28.50 and of Llama-3.2-1B-Instruct from 13.76 to 14.12, with mixed per-task changes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate classifier-head weights with D2NWG instead of meta-learning | Conv4 few-shot classifier · mini-ImageNet 5-way 1-shot | accuracy: $56.41 \pm 0.80\%$ -> 61.13 ± 8.50% | Helped |
| Generate classifier-head weights with D2NWG instead of MetaDiff | Conv4 few-shot classifier · mini-ImageNet 5-way 5-shot | accuracy: 73.18 $\pm$ 0.64$\%$ -> 76.94 ± 6.04% | Helped |
| Use CLIP image embeddings for dataset encoding (ablation) | D2NWG with ResNet18 · CIFAR-10 zero-shot | accuracy: 33.03 $\pm$ 0.04 -> 60.42± 0.75 | Helped |
| Use CLIP image embeddings for dataset encoding (ablation) | D2NWG with ResNet18 · STL-10 zero-shot | accuracy: 50.42 $\pm$ 0.13 -> 82.42 ± 0.04 | Helped |
| Generate task-conditioned LoRA parameters with D2NWG | RoBERTa-Base · GLUE average | average score: 85.2 -> 85.3 | Helped |
| Sample task-specific weights | Llama-3.1-8B-Instruct · Winogrande 5-shot | score: 67.17 -> 67.61 | Helped |
| Sample task-specific weights | Mistral-7B-Instruct · ARC-Challenge 25-shot | score: 59.22 -> 59.80 | Helped |
| Sample task-specific weights | Llama-3.2-1B-Instruct · HellaSwag 25-shot | score: 61.67 -> 61.70 | Helped |
| Sample weights from pretrained LLM parameter distribution | Meta-Llama-3.2-1B-Instruct · GPQA | score: 3.36 -> 1.68 | Hurt |
| Sample weights from pretrained LLM parameter distribution | Meta-Llama-3.2-1B-Instruct · Open LLM leaderboard average | average score: 13.76 -> 14.12 | Helped |
| Sample weights from pretrained LLM parameter distribution | Meta-Llama-3.1-8B-Instruct · Open LLM leaderboard average | average score: 27.91 -> 28.50 | Helped |
| Increase the model zoo from 5,000 to 10,000 ImageNet subsets (ablation) | D2NWG · CIFAR-10 | accuracy: $39.60\pm1.31\%$ -> $42.15\pm2.12$ | Helped |
Surprising findings
- Generated weights on LLM tasks show mixed per-benchmark changes, with some tasks (e.g., GPQA, MuSR) dropping while others rise.
Limitations
- Relies on large collections of pretrained weight tensors and datasets, requiring substantial storage and compute.
- Focuses mainly on cross-dataset generalization; memory constraints, convergence challenges and weight-space symmetries remain open.
Experimental setup
Datasets: ImageNet-1k mini-ImageNet tiered-ImageNet CIFAR-10 CIFAR-100 STL-10 GLUE Meta-Album MNIST SVHN
Benchmarks: mini-ImageNet tiered-ImageNet CIFAR-10 STL-10 Aircraft Pets CIFAR-100 GLUE Winogrande ARC-Challenge HellaSwag Open LLM Leaderboard v2
Metrics: accuracy MCC Pearson correlation
Baselines: iMAML ALFA COMNL MetaQDA MetaDiff MetaOptNet GHN2 GHN3
Builds on: latent diffusion models DDPM Set Transformer generative hyper-representation learning GHN3
Closest papers by meaning
- Weight Diffusion for Future: Learn to Generalize in Non-Stationary Environments (NeurIPS 2024 · similarity 0.87)
- Transfer Learning for Diffusion Models (NeurIPS 2024 · similarity 0.86)
- Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation (ICML 2025 · similarity 0.86)
- Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting (NeurIPS 2024 · similarity 0.85)
- DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models (ICLR 2024 · similarity 0.85)
- Initializing Models with Larger Ones (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.