Research map / Large language models / Language, safety and interpretability

Stable Anisotropic Regularization

ICLR 2024 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IsoScore*, a differentiable, mini-batch-stable isotropy measure, and I-STAR, a regularizer built on it, finding that decreasing isotropy of fine-tuned BERT-family encoders improves GLUE/SQuAD performance.

Problem

Prior NLP claims that higher isotropy of contextual embeddings improves performance rest on flawed isotropy measures (average cosine similarity, partition score), and existing accurate measure IsoScore is non-differentiable and unstable on small mini-batches, so it cannot regularize models.

Approach

IsoScore* makes IsoScore differentiable and stable by replacing the sample covariance with an RDA-shrinkage covariance estimate pooled from a larger reference sample; I-STAR adds (1 - IsoScore*) of all-layer token embeddings as a loss term, with a signed weight to increase or decrease isotropy.</254>

Key results

• ALBERT I-STAR averages on the nine tasks exceed ALBERT Base on most tasks, e.g. RTE 72.56 vs 71.34 and COLA 55.22 vs 54.56 (tuned lambda negative).
• BERT I-STAR improves over BERT Base on RTE (62.53 vs 62.24) and COLA (59.60 vs 59.22), with mixed results on other tasks.
• CosReg measured IsoScore* values near 0 (0.004-0.007) after fine-tuning on SST-2, indicating it does not raise isotropy; setting zeta=0 drops performance by up to 6.19% for BERT.

Measured effects reported in the paper

Surprising findings

• Decreasing isotropy (negative lambda) improved downstream fine-tuning performance, contrary to prior NLP claims that isotropy helps.
• Average cosine similarity based CosReg does not actually increase isotropy as measured by IsoScore*.

Limitations

• Study limited to fine-tuning small encoder models, not LLM pre-training.
• Results for ALBERT on MRPC and COLA show no clear isotropy-performance correlation due to noise.
• Shrinkage is required; without it performance can drop substantially.

Experimental setup

Datasets: SST-2 QNLI RTE MRPC QQP COLA STS-B SST-5 SQUAD

Benchmarks: SST-2 QNLI RTE MRPC QQP COLA STS-B SST-5 SQUAD

Metrics: accuracy Matthews correlation Pearson correlation F1/EM

Baselines: CosReg Base fine-tuning without regularization IsoScore (zeta=0)

Builds on: IsoScore CosReg regularized discriminant analysis TwoNN

Closest papers by meaning

• Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.85)
• SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization (NeurIPS 2026 · similarity 0.84)
• Repetition Improves Language Model Embeddings (ICLR 2025 · similarity 0.84)
• Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.83)
• Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.83)
• Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.83)