Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction
Konstantin Schürholt, Dimche Kostadinov, Damian Borth
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add permutation augmentation (ablation) | EcD · TETRIS-SEED | test reconstruction R2 (%): 49.2 -> 72.9 | Improved |
| add permutation to erasing and noise augmentation (ablation) | EcD · TETRIS-SEED | test reconstruction R2 (%): 65.4 -> 84.1 | Improved |
| add erasing to permutation augmentation (ablation) | EcD · TETRIS-SEED | test reconstruction R2 (%): 72.9 -> 84.1 | Improved |
| add noise augmentation to permutation and erasing (ablation) | EcD · TETRIS-SEED | test reconstruction R2 (%): 84.1 -> 84.1 | No clear effect |
| combine reconstruction with contrastive loss (EcD vs ED) (ablation) | hyper-representation · TETRIS-SEED | accuracy prediction R2 (%): 68.9 -> 90.2 | Improved |
| add reconstruction to contrastive loss (EcD vs Ec) (ablation) | hyper-representation · TETRIS-SEED | accuracy prediction R2 (%): 90.8 -> 90.2 | No clear effect |
| use EcD instead of Ec+D (ablation) | hyper-representation · TETRIS-SEED | accuracy prediction R2 (%): 69.9 -> 90.2 | Improved |
| switch feed-forward to attention on neuron tokens with compression token (ablation) | AttN+t · TETRIS-SEED | accuracy prediction R2 (%): 85.3 -> 90.2 | Improved |
| tokenize each weight (AttW) instead of per neuron (AttN) (ablation) | attention encoder · TETRIS-SEED | reconstruction R2 (%): 89.4 -> 6.8 | Hurt |
| add compression token to weight-token attention (ablation) | AttW · TETRIS-SEED | reconstruction R2 (%): 6.8 -> 74.1 | Improved |
| add compression token to neuron-token attention (ablation) | AttN · TETRIS-SEED | accuracy prediction R2 (%): 88.4 -> 90.2 | Improved |
| add compression token to neuron-token attention (ablation) | AttN · TETRIS-SEED | reconstruction R2 (%): 89.4 -> 84.1 | Hurt |
Closest papers by meaning
- Understanding Representation Learnability of Nonlinear Self-Supervised Learning (AAAI 2023 · similarity 0.89)
- Improving Self-Supervised Learning by Characterizing Idealized Representations (NeurIPS 2022 · similarity 0.87)
- Non-parametric Representation Learning with Kernels (AAAI 2024 · similarity 0.87)
- Towards Scalable and Versatile Weight Space Learning (ICML 2024 · similarity 0.87)
- Self-supervised Representation Learning from Random Data Projectors (ICLR 2024 · similarity 0.86)
- Measuring Self-Supervised Representation Quality for Downstream Classification Using Discriminative Features (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
