Research map / Architectures and efficiency / Pruning and efficient tuning
Unsupervised Model Tree Heritage Recovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Unsupervised MoTHer Recovery, which reconstructs Model Trees (heritage graphs) from weights alone using weight distance, a kurtosis-based directional score, and a minimum directed spanning tree.
Problem
Model heritage is undocumented on model hubs (over 60% of Hugging Face models lack heritage info in their cards), creating attribution and IP problems; the parent-child structure of fine-tuned model collections is unknown.
Approach
Weight distance (full-weight L2 or LoRA-rank-based) identifies which model pairs are directly related, and kurtosis-based outlier counts change monotonically between generalization and specialization training, giving edge direction; a directed minimum spanning tree over the combined matrix recovers the tree without data access.
Key results
- On the MoTHer LoRA-V sub-graph, MoTHer reconstructs all 5 Model Trees with accuracy 1.0; the FT sub-graph reaches a Model Graph accuracy of 0.94 and the Mixed sub-graph 0.83.
- On the Stable Diffusion Model Tree, MoTHer recovers all but one edge (SD 1.4 placed as child of 1.3 instead of sibling).
- With 99% of weights pruned, accuracy remains 68% versus roughly 5% random baseline; 50% quantized models lower performance by less than 1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LoRA rank distance ℓLoRA instead of ℓFT for LoRA fine-tuned models (ablation) | MoTHer · LoRA-V Model Graph | accuracy: 0.78 -> perfect accuracy | Helped |
| increase tree size and depth to a 121-model, 5-level hierarchy (ablation) | MoTHer · Deep ViT Model Tree | accuracy: 0.85 to 0.79 | Hurt |
| prune 90% of model weights (ablation) | MoTHer · Pruned ViT Model Graph | accuracy: 0.89 -> 0.85 | Hurt |
| prune 99% of model weights (ablation) | MoTHer · Pruned ViT Model Graph | accuracy: 0.89 -> 0.68 | Hurt |
| quantize 50% of models to fp16 (ablation) | MoTHer · ViT Model Graph | accuracy: 0.89 -> 0.883 ± 0.0044 | Hurt |
| quantize 50% of models to Int8 (ablation) | MoTHer · ViT Model Graph | accuracy: 0.89 -> 0.883 ± 0.0078 | Hurt |
| use varying LoRA ranks rather than a fixed rank | MoTHer · LoRA Model Graph recovery | accuracy: 1.0 vs 0.96 | Helped |
| use kurtosis rather than variance, skewness, or entropy as the directional score (ablation) | MoTHer · Directional score analysis | monotonicity | Helped |
| use a dense layer for the directional score (ablation) | MoTHer · Fine-tuning across Model Trees | directional-score consistency | Helped |
| quantize half of the models (ablation) | MoTHer · ViT Model Graph | accuracy: less than 1% | Hurt |
| recover a tree with 3 same-recipe ViT models using different seeds (ablation) | MoTHer · ViT Model Tree with similar models | accuracy | Helped |
| increase pruning to 95% (ablation) | MoTHer · Pruned ViT Model Graph | accuracy: 0.89 -> 0.8 | Hurt |
Surprising findings
- Using only the weights, ViT models sharing the same architecture but unrelated (DINO, MAE, CLIP) can be separated into distinct trees by weight distance.
- Kurtosis was the only candidate directional score (vs. variance, skewness, entropy) that showed consistent monotonicity across models.
Limitations
- Assumes the training stage (generalization vs. specialization) of each node is known in advance.
- Handles only single-parent heritage; merged models are only preliminarily explored.
- Web-scale recovery is not tested; pairwise distance computation is costly at millions of models.
Experimental setup
Datasets: VTAB CIFAR-100 ImageNet
Benchmarks: MoTHer dataset Stable Diffusion Model Tree (Hugging Face)
Metrics: edge accuracy (correct edge placement and direction) clustering accuracy runtime
Baselines: ℓ_FT weight distance (ablation for LoRA models) variance, skewness, entropy (alternative directional scores)
Builds on: LoRA Chu-Liu-Edmonds algorithm Model Soups
Closest papers by meaning
- Learning on Model Weights using Tree Experts (CVPR 2025 · similarity 0.84)
- On the Role of Edge Dependency in Graph Generative Models (ICML 2024 · similarity 0.83)
- Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective (ICML 2024 · similarity 0.83)
- Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models (ICLR 2024 · similarity 0.83)
- Machine Unlearning for Image-to-Image Generative Models (ICLR 2024 · similarity 0.82)
- ModHiFi: Identifying High Fidelity predictive components for Model Modification (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.