Research map / Label-efficient and robust learning / Recognition and long-tail

Invariant Features in Language Models: Geometric Characterization and Model Attribution

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Multilingual language representation pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a local geometric account of semantic invariance in LLM hidden states, a contrastive generalized-eigenvalue subspace method separating paraphrase vs semantic-changing variation, and uses the invariant signatures for zero-shot model attribution across nine LLMs.

Problem

Language models are robust to paraphrasing, but it is unclear where semantic invariance arises across depth, how to formally characterize it geometrically, and how it differs across models.

Approach

Identify invariant directions per layer as the generalized eigenvectors of semantic-changing sensitivity covariance relative to semantic-preserving (paraphrase) sensitivity covariance, then project hidden states onto this subspace and use the projections as model signatures for zero-shot attribution.

Key results

• Nuisance projection energy for semantic displacement is about 0.01-0.02 across nine models and three groups, indicating semantic displacement lies largely outside the nuisance subspace.
• Zero-shot attribution accuracy for base models is typically above 92% (e.g., Llama3 94.1% in Group1), dropping about 2-4 points under fine-tuning and distillation.
• In the PCA comparison, the invariant representation attains 96.8% attribution on Llama-3.1-8B versus 87.4% for PCA on base models.

Surprising findings

• Nuisance-subspace swaps produce non-negligible output shifts even for semantic-preserving donors, which the authors attribute to residual semantic content in the nuisance component.

Limitations

• Nuisance subspace is not cleanly disentangled; interventions on it still change outputs for SP donors.
• Small self-constructed dataset of 410 prompts across 10 groups; invariant zones are local per group.
• Layer selection and invariant zones are chosen per group and model with limited theoretical justification; claims of geometric manifold structure are largely heuristic.

Experimental setup

Datasets: self-constructed 410-prompt paraphrase/semantic-change dataset

Benchmarks: self-constructed 410-prompt dataset

Metrics: generalized eigenvalue ratio nuisance projection energy KL divergence MSE attribution accuracy

Baselines: PCA projection

Builds on: manifold hypothesis centered kernel alignment prototypical networks

Closest papers by meaning

• Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.88)
• From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication (ICLR 2024 · similarity 0.87)
• The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities (ICLR 2025 · similarity 0.87)
• DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models (ICLR 2025 · similarity 0.87)
• The Geometry of Categorical and Hierarchical Concepts in Large Language Models (ICLR 2025 · similarity 0.86)
• The Linear Representation Hypothesis and the Geometry of Large Language Models (ICML 2024 · similarity 0.86)