Research map / Architectures and efficiency / RNNs and normalization
Independence Tests for Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes permutation-based statistical tests (constrained: exact p-values; unconstrained: MATCH test over aligned GLU MLP activations) to determine whether two open-weight LMs were trained independently from their weights.
Problem
Given only the weights of two models, test whether they were trained independently from random initializations (versus one derived from the other via fine-tuning, pruning, or sharing), for provenance and IP/liability auditing.
Approach
Under permutation-equivariant training and permutation-invariant initialization, simulate exchangeable copies of a model by permuting hidden units to obtain exact p-values (constrained setting). For unconstrained settings, learn GLU MLP proxies that match arbitrary submodels and correlate gate/up activation alignments (MATCH), which behaves empirically like a p-value and enables localized testing.
Key results
- In the constrained setting, the Csw and Csh statistics yield p-values below 2.2e-308 for all 69 non-independent pairs among 21 Llama-architecture models, while independent pairs are uniform.
- The unconstrained MATCH statistic is near-uniform on 141 independent pairs and below 2.2e-308 for non-independent pairs, including when a single MLP block is retrained from scratch.
- MATCH localizes pruning: it identifies the Llama-3.1-8B blocks that map onto Llama-3.2-3B and Llama-3.2-1B layers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CSW or CSH instead of raw Jensen-Shannon divergence for constrained independence testing | Llama-7B model pairs · Llama-7B model pairs | independence-test p-value: ε | Helped |
| Use φ_l2 with permutation-test p-values rather than raw ℓ2 distance | Constrained independence testing · Llama-7B model pairs | independence-test p-value: 0.01 | Helped |
| Use Jensen-Shannon divergence as the independence statistic | Llama-7B model pairs · Llama-7B model pairs | Jensen-Shannon divergence | No clear effect |
| Use φ_MATCH for independently initialized models trained with the same algorithm and dataset (ablation) | Unconstrained independence testing · OLMo 7B on Dolma v1_7 checkpoints | φ_MATCH p-value: 0.809; 0.443; 0.343; 0.027 | No clear effect |
| Retrain an individual MLP layer from scratch before applying φ_MATCH (ablation) | vicuna-7b-v1.5 compared with Llama-2-7b-hf · All 32 MLP layers | log10(φ_MATCH^(i)): {-479}; {-485}; {-614}; {-580}; {-523}; {-513}; {-533}; {-464}; {-439}; {-377}; {-365}; {-342}; {-330}; {-323}; {-414}; | No clear effect |
| Distill non-GLU model layers to GLU MLP proxies before applying φ_MATCH (ablation) | Generalized robust test · GPT2_PMC vs gpt2 | φ_MATCH^(1) p-value: 3.034e-61 | Helped |
| Test model pairs with different architectures using activation matching | φ_MATCH · Different-architecture dependent pairs | φ_MATCH^(1) p-value: 3.034e-61; 1.049e-75; 1.079e-63; 2.011e-70 | Helped |
| Test independent model pairs with φ_MATCH after fitting GLU MLP proxies | Generalized robust test · Independent GPT and Llama model pairs | φ_MATCH^(1) p-value: 0.359; 0.770; 0.481 | No clear effect |
| Apply φ_MATCH to retrained MLP layers rather than original layer weights (ablation) | Unconstrained test on dependent model pairs · vicuna-7b-v1.5 vs Llama-2-7b-hf | φ_MATCH p-value: less than ε | No clear effect |
| Use φ_MATCH for localized block matching after pruning | Llama 3.1-8B and Llama 3.2-3B/1B · Pruned Llama model pairs | block-match statistic threshold: 1e-4 | Helped |
| Reduce both hidden and MLP dimensions through pruning and apply φ_MATCH | Sheared-LLaMa-2.7B vs Llama-2-7B · 32 Transformer blocks | φ_MATCH^(i,j) threshold: 1e-90 | Helped |
Surprising findings
- Output-distribution JSD does not reliably separate dependent from independent pairs; CodeLlama-7b-hf diverges more from Llama-2-7b-hf than unrelated independently trained models do.
- The l2 distance between weights is not reliable on raw values, but standardizing via permutation p-values makes it informative.
Limitations
- Constrained setting is not robust to permutation or architecture changes, so adversarial transforms break it.
- Unconstrained MATCH does not yield provably exact p-values.
- Does not distinguish between different dependency types (e.g., shared ancestor vs fine-tune).
Experimental setup
Datasets: WikiText-103 Dolma v1_7
Benchmarks: 21 Llama-7B-architecture open-weight model pairs (210 pairs) 141 independent model pairs (unconstrained) OLMo checkpoint pairs
Metrics: p-value Spearman correlation uniformity of p-values under null
Baselines: phi_l2 weight distance (Oyama et al.) JSD on output distributions Oyama et al. invariant tests
Builds on: Fisher's method Jensen-Shannon divergence baseline Spearman rank correlation
Closest papers by meaning
- Blackbox Model Provenance via Palimpsestic Membership Inference (NeurIPS 2025 · similarity 0.88)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.87)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.86)
- SimpleStrat: Diversifying Language Model Generation with Stratification (NeurIPS 2025 · similarity 0.86)
- Understanding Emergent Abilities of Language Models from the Loss Perspective (NeurIPS 2024 · similarity 0.85)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.