Research map / Architectures and efficiency / Pruning and efficient tuning
Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Unextractable Protocol Models (UPMs), which morph pipeline-stage weights with random invertible transforms that cancel within a time step so that full weights can never be materialized or stitched across time steps.
Problem
In decentralized collaborative training and serving of large models, participants who hold model shards over time can stitch them together (piecewise Sybil attacks) and extract the full weights, undermining economic viability.
Approach
Periodically fold ephemeral random invertible transforms (Haar orthogonal or low-condition matrices) into adjacent pipeline-stage weights so the end-to-end function is preserved at each time step but weights from different time steps are mutually incompatible; combined with skip connections for full incompatibility and Muon-compatible training.
Key results
- Orthogonal transforms match the no-transform baseline on GPT-2 (val loss 3.27 vs 3.28 original) and GPT-2 Med/XL when training with Muon on FineWeb.
- Inference morphing with high-precision workaround keeps logit drift and perplexity negligible over 10k morphing steps on Qwen 2.5-0.5B and Llama 3.2-1B.
- Stitched-model learning-based attacks still required at least ~60% of the compute of training from scratch to reach perplexity 20; overhead is ~3% latency and ~0.1% bandwidth amortized.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply orthogonal transforms during training | UPM with Muon · FineWeb validation across GPT-2, GPT-2 Med, and GPT-2 XL | validation loss: 3.28 -> 3.27; 3.26 -> 3.26; 3.16 -> 3.17 | No clear effect |
| Use low-condition number transforms with ε = 0.01 (ablation) | UPM with Muon · FineWeb validation across GPT-2, GPT-2 Med, and GPT-2 XL | validation loss: 3.28; 3.26; 3.16 | No clear effect |
| Use low-condition number transforms with ε = 0.1 (ablation) | UPM with Muon · FineWeb validation across GPT-2, GPT-2 Med, and GPT-2 XL | validation loss: 3.66; 3.74; 4.48 | Hurt |
| Use low-condition number transforms with ε = 0.5 (ablation) | UPM with Muon · FineWeb validation across GPT-2, GPT-2 Med, and GPT-2 XL | validation loss: 5.09; 5.01; 5.13 | Hurt |
| Use high-precision weights on disk before casting to low precision (ablation) | UPM inference · WikiText (v2-raw) validation | perplexity | Helped |
| Use high-precision weights on disk before casting to low precision (ablation) | UPM inference · WikiText (v2-raw) validation | Jensen-Shannon distance | Helped |
| Apply UPM morphing with orthogonal transforms | Llama 3.2 1B · FineWeb learning-based attack | compute as percentage of training from scratch: roughly 60% | Hurt |
Surprising findings
- Without a high-precision accumulation workaround, FP16 and BF16 inference with repeated morphing drift substantially and eventually render the model unusable, even though FP32 stays stable.
- Learning-based attacks on stitched models remain costly (>=60% of from-scratch compute), so stitching across time steps does not cheaply recover a functional model.
Limitations
- Security relies on an honest majority of participants; security weakens if a large fraction is dishonest.
- Experiments are single-machine simulations and exclude real decentralized deployment and side-channel attacks (timing, cache).
- Broader impact: unextractable weights prevent safety inspection and content moderation.
Experimental setup
Datasets: FineWeb WikiText-2 (raw)
Benchmarks: WikiText-2 (raw) validation perplexity FineWeb validation loss
Metrics: Jensen-Shannon distance of logits perplexity validation loss relative loss gap tokens-to-perplexity relative to training from scratch latency overhead bandwidth overhead memory overhead
Baselines: no-transform baseline Muon unconstrained training
Builds on: Muon Model Parallelism Pipeline Parallelism RMSNorm SWARM
Closest papers by meaning
- CoRe: Collaborative Reasoning via Cross Teaching (ICML 2026 · similarity 0.85)
- Occult: Optimizing Collaborative Communications across Experts for Accelerated Parallel MoE Training and Inference (ICML 2025 · similarity 0.85)
- Decentralized Diffusion Models (CVPR 2025 · similarity 0.84)
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.84)
- DCHM: Dynamic Collaboration of Heterogeneous Models Through Isomerism Learning in a Blockchain-Powered Federated Learning Framework (AAAI 2025 · similarity 0.84)
- PDUDT: Provable Decentralized Unlearning under Dynamic Topologies (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.