Research map / Label-efficient and robust learning / Self-supervised and contrastive
Improved Generalization of Weight Space Networks via Augmentations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes overfitting in deep weight space networks on INR datasets and proposes weight-space augmentations, including alignment-based weight-space MixUp, which improve DWS and GNN classifiers and SimCLR pretraining.
Problem
Deep weight space (DWS) models that process weights of implicit neural representations (INRs) overfit heavily, largely because datasets contain only one neural view per object, limiting diversity.
Approach
Categorizes weight-space augmentations (input-space, data-agnostic, and network-specific symmetry-based such as SIREN negation/bias) and proposes MixUp for weight spaces that first aligns permutation symmetries (via Weight Matching) before interpolating weights and labels.
Key results
- Alignment + MixUp with DWS on a single view reaches 27.40% on ModelNet40-INR vs 16.17% without augmentation, and 75.67% vs 68.30% on FMNIST-INR.
- Augmentation with a single view is claimed comparable to training with 5-10x more data across INR datasets.
- SimCLR pretraining with weight-space augmentations improves DWS and GNN by up to ~11% and ~8% accuracy in semi-supervised settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add weight space MixUp | DWS · ModelNet40-INR, 1 View | test accuracy: 16.17 \pm 0.25 -> 26.96 \pm 0.91 | Helped |
| Align weights before MixUp | DWS · FMNIST-INR | test accuracy: 68.30 \pm 0.62 -> 75.67 \pm 0.36 (1 View); 76.01 \pm 1.20 -> 79.41 \pm 0.56 (10 View) | Helped |
| Align weights before MixUp | GNN · ModelNet40-INR | test accuracy: 8.82 \pm 1.08 -> 26.88 \pm 1.75 (1 View); 34.51 \pm 1.24 -> 42.83 \pm 4.18 (10 View) | Helped |
| Use random permutation before MixUp instead of learned weight alignment | DWS · CIFAR10-INR, 1 View | test accuracy: 42.76 \pm 0.12 -> 38.34 \pm 0.54 | Hurt |
| Add SIREN negation augmentation | DWS · CIFAR10-INR | test accuracy: 39.42 \pm 1.11 -> 31.03 \pm 0.66 (1 View); 40.64 \pm 0.22 -> 29.15 \pm 0.67 (5 View) | Hurt |
| Add masking augmentation | DWS · ModelNet40-INR | test accuracy: 16.17 \pm 0.25 -> 11.43 \pm 2.44 (1 View); 30.25 \pm 0.95 -> 14.71 \pm 1.14 (10 View) | Hurt |
| Add contrastive pre-training with weight space augmentations | DWS · FMNIST, 5% labeled | test accuracy: 48.55 \pm 0.59 -> 59.49 \pm 0.69 | Helped |
| Add contrastive pre-training with weight space augmentations | GNN · CIFAR10, 5% labeled | test accuracy: 21.07 \pm 1.29 -> 26.93 \pm 1.08 | Helped |
| Add label smoothing without input averaging (ablation) | DWS MixUp · FMNIST-INR, 1 View | test accuracy: 68.30 \pm 0.62 -> 69.15 \pm 0.51 | Helped |
| Add input averaging without label smoothing (ablation) | DWS MixUp · FMNIST-INR, 1 View | test accuracy: 68.30 \pm 0.62 -> 72.38 \pm 0.39 | Helped |
| Combine label smoothing and input averaging as MixUp (ablation) | DWS · FMNIST-INR, 1 View | test accuracy: 72.38 \pm 0.39 -> 74.36 \pm 1.17 | Helped |
| Use feature MixUp instead of weight space MixUp | DWS · FMNIST-INR | test accuracy: 68.47 | Hurt |
Surprising findings
- Alignment-based MixUp only modestly outperforms direct MixUp and random-permutation MixUp, which the authors attribute to suboptimal alignment that ignores activation symmetries.
- Restricting model capacity does not reduce overfitting; the most expressive DWS variant had the best test performance.
Limitations
- Gap remains compared to directly processing images or point clouds (e.g., ~16% vs ~90% on ModelNet40).
- Alignment only considers permutation symmetries and ignores activation (negation) symmetries.
- Evaluation focuses on INR datasets and classification; other tasks such as generalization prediction are not evaluated.
Experimental setup
Datasets: ModelNet40 FMNIST CIFAR10
Benchmarks: ModelNet40-INR FMNIST-INR CIFAR10-INR
Metrics: test accuracy
Baselines: DWS without augmentation GNN without augmentation Direct MixUp MixUp + random permutation Translate SIREN negation Masking Gaussian noise Feature MixUp
Builds on: DWS MixUp Weight Matching SimCLR SIREN
Closest papers by meaning
- Towards Scalable and Versatile Weight Space Learning (ICML 2024 · similarity 0.85)
- How Data Augmentation Shapes Neural Representations (NeurIPS 2026 · similarity 0.84)
- Function-space Parameterization of Neural Networks for Sequential Learning (ICLR 2024 · similarity 0.84)
- Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks (NeurIPS 2025 · similarity 0.84)
- Deep Linear Probe Generators for Weight Space Learning (ICLR 2025 · similarity 0.84)
- Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.