Research map / Large language models / Language, safety and interpretability
DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DOCS, a Gumbel-fitted distribution-of-maximum-cosine-similarity index for comparing LLM weight matrices, shown to discriminate between orthogonal matrices, and used to reveal adjacent-layer similarity and layer clusters in open LLMs.
Problem
Existing representation-similarity indices (CCA, SVCCA, linear CKA, HSIC) are not suited to comparing LLM weight matrices: they are constant or dimension-dependent on orthogonal matrices, and representation similarity does not imply weight similarity due to residual connections.
Approach
DOCS computes the absolute cosine similarity between each column of one weight matrix and its best-matching column in the other, fits Gumbel distributions to these maxima for both directions, and averages the location parameters, so it can distinguish orthogonal matrices and captures strong per-neuron alignments.
Key results
- DOCS achieves the highest Gini coefficient (0.0745) on MLP-Up of Llama-3.1-8B-Instruct among the compared indices, versus 0.0488 for linear CKA.
- Adjacent layers show high DOCS similarity across models, with similarity decreasing as layer distance grows; base vs. instruct DOCS scores exceed 0.7 for every matrix evaluated.
- Layer clusters appear in the middle depths and final layers of gemma-2-9b, gemma-2-27b, Llama-3.1-8B and Mixtral-8x7B; the paper proves DOCS yields 1/sqrt(m) on a constructed pair of orthogonal matrices where Frobenius distance is Omega(sqrt(m)).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DOCS instead of the evaluated similarity indices to compute weight-similarity heatmaps | MLP-Up layers of Meta-Llama-3.1-8B-Instruct · MLP-Up, Meta-Llama-3.1-8B-Instruct | Gini coefficient | Helped |
| Compare adjacent transformer layers with farther-apart layers using DOCS | Open-source LLM weight matrices · Transformer layers across open-source LLMs | DOCS similarity | Mixed |
| Compare base and instruction-tuned weights using DOCS | yi-1.5, Llama-3.1, and gemma-2 · Base versus instruction-tuned model weight matrices | DOCS similarity | No clear effect |
Surprising findings
- The first and last layers, the most distant pair, also show higher DOCS similarity, which the authors attribute to their connection to token embeddings.
- In MoE models, typically one expert (e.g., the third) stands out as dissimilar from the others in many layers.
Limitations
- Cluster-based applications (PEFT inductive biases, sparsity, distillation) are proposed but not experimentally validated.
- Gumbel fitting and transposition conventions are design choices; the Gini comparison is the only quantitative benchmark against other indices.
Experimental setup
Metrics: DOCS similarity index Gini coefficient
Baselines: Linear Regression CCA SVCCA Linear HSIC Linear CKA
Builds on: Canonical Correlation Analysis (CCA) SVCCA linear CKA Hilbert-Schmidt Independence Criterion (HSIC)
Closest papers by meaning
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.87)
- Invariant Features in Language Models: Geometric Characterization and Model Attribution (NeurIPS 2026 · similarity 0.87)
- Correlation Dimension of Autoregressive Large Language Models (NeurIPS 2025 · similarity 0.86)
- Exploring Activation Patterns of Parameters in Language Models (AAAI 2025 · similarity 0.86)
- Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.86)
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.