Research map / Architectures and efficiency / RNNs and normalization
Learning on Model Weights using Tree Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that models within a Model Tree share less nuisance variation, and proposes ProbeX, a lightweight single-layer probing expert per tree that predicts training classes and aligns Stable Diffusion LoRA weights to CLIP text for zero-shot model classification.
Problem
Inferring undocumented model functionality (training data categories) directly from model weights is hard because weights contain large nuisance variation; dense weight-space metanetworks are too parameter-heavy for large models.
Approach
Group models into Model Trees (sharing a common ancestor) so nuisance variation is reduced, learn one expert per tree, and make the expert a lightweight probing network on a single hidden layer (learned probes, shared projection, per-probe encoders) that factorizes a Tucker-style dense linear expert.
Key results
- On the discriminative split, ProbeX reaches 0.842 (ResNet), 0.705 (DINO), 0.765 (MAE), 0.885 (Sup. ViT) accuracy versus 0.713/0.614/0.666/0.663 for the dense expert, with about 2.3M parameters versus 59-105M.
- On SD200 held-out classes, ProbeX achieves 0.898 zero-shot accuracy with 30 held-out classes, versus 0.706 for the dense expert.
- ReLU non-linearity raises SD200 zero-shot accuracy from 0.564 to 0.898, while in-distribution accuracy rises from 0.953 to 0.973.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ProbeX instead of the Dense expert | Model-J discriminative class prediction · Model-J discriminative, ResNet | accuracy: 0.713 -> 0.842 | Helped |
| Use ProbeX instead of the Dense expert | Model-J discriminative class prediction · Model-J discriminative, DINO | accuracy: 0.614 -> 0.705 | Helped |
| Use ProbeX instead of the Dense expert | Model-J discriminative class prediction · Model-J discriminative, MAE | accuracy: 0.666 -> 0.765 | Helped |
| Use ProbeX instead of the Dense expert | Model-J discriminative class prediction · Model-J discriminative, Sup. ViT | accuracy: 0.663 -> 0.885 | Helped |
| Use ProbeX instead of the Dense expert | Model-J discriminative class prediction · Model-J discriminative, MoE | accuracy: 0.664 -> 0.799 | Helped |
| Use ProbeX instead of the Dense expert for text-aligned model classification | Stable Diffusion weight-text alignment · SD200, zero-shot | accuracy: 0.706 -> 0.898 | Helped |
| Use ProbeX instead of the Dense expert for text-aligned model classification | Stable Diffusion weight-text alignment · SD1k, zero-shot | accuracy: 0.343 -> 0.505 | Helped |
| Use ProbeX instead of the Dense expert for text-aligned model classification | Stable Diffusion weight-text alignment · SD1k, in-distribution | accuracy: 0.382 -> 0.296 | Hurt |
| Add ReLU activation to ProbeX (ablation) | Stable Diffusion weight-text alignment · SD200, zero-shot | accuracy: 0.564 -> 0.898 | Helped |
| Add ReLU activation to ProbeX (ablation) | Stable Diffusion weight-text alignment · SD200, in-distribution | accuracy: 0.953 -> 0.973 | Helped |
| Use CLIP rather than OpenCLIP as the text encoder (ablation) | ProbeX weight-text alignment · SD200, zero-shot | accuracy: 0.860 -> 0.898 | Helped |
| Use CLIP rather than BLIP2 as the text encoder (ablation) | ProbeX weight-text alignment · SD200, zero-shot | accuracy: 0.564 -> 0.898 | Helped |
Surprising findings
- Adding models from other Model Trees to training degrades accuracy on a single tree (negative transfer), while within-tree data helps.
- A linear classifier on a single layer works well within a Model Tree but near random across many trees, even with a shared single epoch of pre-training as the only difference.
Limitations
- Cannot generalize to new Model Trees without training a new expert for each tree.
- Zero-shot alignment works for SD models but not for discriminative-split models, which generalize only in-distribution.
- Deeper probe encoders reduced performance, likely due to overfitting.
Experimental setup
Datasets: CIFAR100 ImageNet Model-J (Model Jungle)
Benchmarks: Model-J
Metrics: accuracy zero-shot accuracy kNN accuracy mean ROC AUC (one-class classification) parameter count
Baselines: StatNN Dense Expert Neural Graphs Random Raw weights kNN
Builds on: Model Trees (Horwitz et al. 2024) Neural Graphs StatNN (Unterthiner et al.) CLIP
Closest papers by meaning
- Deep Linear Probe Generators for Weight Space Learning (ICLR 2025 · similarity 0.84)
- Unsupervised Model Tree Heritage Recovery (ICLR 2025 · similarity 0.84)
- Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training (ICML 2026 · similarity 0.83)
- Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models (NeurIPS 2025 · similarity 0.83)
- Structured Model Probing: Empowering Efficient Transfer Learning by Structured Regularization (CVPR 2024 · similarity 0.82)
- Automated Statistical Model Discovery with Language Models (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.