Research map / Large language models / Language, safety and interpretability
On Affine Homotopy between Language Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies similarity of pre-trained language encoders as functions via asymmetric affine homotopy, proving intrinsic affine alignment bounds extrinsic (downstream) similarity and validating it on ELECTRA, RoBERTa and 25 MultiBERTs.
Problem
Measuring whether two pre-trained language encoders are similar in a task-independent (intrinsic) way that still predicts downstream (extrinsic) performance differences; prior work compares finite representation sets only.
Approach
Defines a hemi-metric on encoder functions via Hausdorff-Hoare lifting of uniform-convergence distance over the orbit of affine transformations, yielding an asymmetric affine intrinsic preorder and a linear bound from intrinsic to extrinsic (log-linear classifier) dissimilarity.
Key results
- Intrinsic affine distance correlates with extrinsic similarity on GLUE tasks; e.g., PCC 0.970 for PWCCA-based intrinsic measure vs. extrinsic on SST-2 (linear alignment measures show strong linear correlation).
- Rank to precision correlates with median intrinsic distance when mapping to an encoder for RTE (rho=0.312), MRPC (rho=0.609), QQP (rho=0.389).
- Affine order is asymmetric: some encoders are consistently easier to map to, and this trend is task-independent for early layers but diverges for later layers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use affine alignment rather than Linear CKA to measure intrinsic similarity | Language encoder representations · SST-2; extrinsic metric: fixed-classifier similarity | Spearman rho: 0.080 vs 0.088 | No clear effect |
| Use affine alignment rather than Linear CKA to measure intrinsic similarity | Language encoder representations · SST-2; extrinsic metric: worst-case classifier similarity | Spearman rho: 0.621 vs 0.295 | Helped |
| Use affine alignment rather than Linear CKA to measure intrinsic similarity | Language encoder representations · MRPC; extrinsic metric: fixed-classifier similarity | Spearman rho: 0.309 vs 0.214 | Helped |
| Use affine alignment rather than Linear CKA to measure intrinsic similarity | Language encoder representations · MRPC; extrinsic metric: worst-case classifier similarity | Spearman rho: 0.231 vs 0.030 | Helped |
| Use affine alignment rather than Linear CKA to measure intrinsic similarity | Language encoder representations · RTE; extrinsic metric: fixed-classifier similarity | Spearman rho: 0.534 vs 0.185 | Helped |
| Use affine alignment rather than Linear CKA to measure intrinsic similarity | Language encoder representations · RTE; extrinsic metric: worst-case classifier similarity | Spearman rho: 0.234 vs 0.240 | No clear effect |
| Use affine alignment rather than Linear CKA to measure intrinsic similarity | Language encoder representations · CoLA; extrinsic metric: fixed-classifier similarity | Spearman rho: 0.196 vs 0.165 | Helped |
| Use affine alignment rather than Linear CKA to measure intrinsic similarity | Language encoder representations · CoLA; extrinsic metric: worst-case classifier similarity | Spearman rho: 0.348 vs 0.380 | Hurt |
Surprising findings
- Affine alignment measure is not a metric and is asymmetric; exact affine homotopy requires equal rank encoders.
- Linear CKA captures extrinsic behavior less faithfully than linear alignment methods.
Limitations
- Affine transformations may not capture non-linear encoder relationships; MultiBERTs are not exactly affinely related.
- Extrinsic analysis restricted to linear classifiers and does not cover full fine-tuning.
- Numerical approximations (max over finite sets, SVD rank tolerance) used in practice.
Experimental setup
Datasets: SST-2 MRPC RTE CoLA MNLI QQP
Benchmarks: SST-2 MRPC RTE CoLA
Metrics: Spearman rank correlation Pearson correlation coefficient p-value
Baselines: Orthogonal Procrustes CCA PWCCA Linear CKA R^2_CCA
Builds on: Hausdorff-Hoare hemi-metric orthogonal Procrustes CCA CKA SVCCA
Closest papers by meaning
- Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures (ICLR 2025 · similarity 0.86)
- Analyzing Similarity Metrics for Data Selection for Language Model Pretraining (NeurIPS 2025 · similarity 0.86)
- Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.86)
- Do Vision and Language Encoders Represent the World Similarly? (CVPR 2024 · similarity 0.86)
- DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models (ICLR 2025 · similarity 0.85)
- Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.