Research map / Large language models / Language, safety and interpretability

Token Embeddings Violate the Manifold Hypothesis

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Linguistic encoding in LLM representations · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a statistical test (manifold and fiber bundle hypotheses) based on log-volume vs log-radius slopes around each point, and applies it to token embeddings of GPT2, Llemma7B, Mistral7B and Pythia6.9B, finding the token subspace frequently is not a manifold or fiber bundle.

Problem

Whether LLM token input embeddings lie on a smooth manifold or fiber bundle, a regularity assumption often tacitly used in analyzing LLMs and dimension reduction methods like UMAP.

Approach

A local-volume-scaling hypothesis test: under the null of a smooth fiber bundle (or manifold), log volume versus log radius should have slopes that do not increase (or stay constant) with radius; rejections at a token indicate singularities in its neighborhood. Implemented via Monte Carlo ball counts, sliding-window t-tests on slopes, and Holm-Bonferroni correction.

Key results

• The manifold hypothesis is rejected for all four LLMs' token embeddings, with 33 to 66 tokens rejecting it (e.g., GPT2 66 tokens, p approx 3e-8).
• The fiber bundle hypothesis is rejected for GPT2 (12 tokens at small radius), Llemma7B (1 at large radius), and Mistral7B (1 small, 2 large), but not for Pythia6.9B.
• Local dimension estimates are highly variable across tokens; e.g., GPT2 small-radius median 389 and large-radius median 14, suggesting context cannot persistently resolve singularities for some models.

Surprising findings

• Some tokens have neighborhoods without a well-defined intrinsic dimension, and tokens with dimension-0 neighborhoods suggest the token subspace is disconnected.
• Llemma7B and Mistral7B share identical token sets yet show significantly different rejection patterns, indicating training leaves model-specific topological fingerprints.

Limitations

• Rejection only implies irregularity; failing to reject is uninformative and the tests do not identify singularity type.
• Only slope changes below the embedded reach are valid rejections; small reach implies high curvature and numerical instability.
• Results depend on chosen v_min/v_max neighborhood sizes selected by inspecting a small sample.

Experimental setup

Benchmarks: synthetic manifolds (sphere, cusp surface, lollipop) token embedding matrices of GPT2, Llemma7B, Mistral7B, Pythia6.9B

Metrics: p-value (Holm-Bonferroni corrected) local dimension quartiles number of rejecting tokens

Baselines: UMAP maximum likelihood intrinsic dimension estimator

Builds on: UMAP manifold hypothesis local intrinsic dimension estimation

Closest papers by meaning

• By Tying Embeddings You Are Assuming the Distributional Hypothesis (ICML 2024 · similarity 0.88)
• Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.88)
• From Tokens to Words: On the Inner Lexicon of LLMs (ICLR 2025 · similarity 0.87)
• Leviathan: Decoupling Input and Output Representations in Language Models (NeurIPS 2026 · similarity 0.86)
• Invariant Features in Language Models: Geometric Characterization and Model Attribution (NeurIPS 2026 · similarity 0.86)
• Hyperbolic Fine-Tuning for Large Language Models (NeurIPS 2025 · similarity 0.86)