Research map / Architectures and efficiency / RNNs and normalization
Persistent Topological Features in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies zigzag persistence from topological data analysis to last-token representations across LLM layers, yielding layer-wise topological descriptors used to define a layer-pruning criterion.
Problem
Existing analyses of LLM internal representations are static per layer and do not track how prompt representations evolve and rearrange across layers; a system-level view of layer dynamics is lacking.
Approach
Treat layers as time steps and build a kNN-based zigzag filtration over last-token embeddings across layers, then derive descriptors (births' relative frequency, inter-layer persistence) from the zigzag persistence diagram, identifying four processing phases and using a plateau-based criterion to prune layers.
Key results
- Layer pruning with the zigzag criterion on Llama 3 keeps MMLU at 53.44 vs 65.07 full, matching the other-works method (53.44) at the same pruned-layer count.
- On WinoGrande, zigzag pruning gives 70.00 for Llama 3 equal to the angular/Bi-score baselines, but is lower for Llama 2 (68.67 vs 67.72 other works, higher) and Mistral (66.50 vs 63.76, higher) and Pythia (55.71 vs 58.09, lower).
- Four phases of prompt processing are identified in Llama 3 8B on SST, with long-lived 1-dimensional holes peaking in middle layers and a sharp rearrangement in the last 2-3 layers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use zigzag-based pruning instead of other-work similarity pruning | Llama 2 · MMLU 5-shot | accuracy: 43.95 -> 37.38 | Hurt |
| Use zigzag-based pruning instead of other-work similarity pruning | Llama 2 · HellaSwag 5-shot | accuracy: 42.78 -> 44.71 | Helped |
| Use zigzag-based pruning instead of other-work similarity pruning | Llama 2 · WinoGrande 5-shot | accuracy: 67.72 -> 68.67 | Helped |
| Use zigzag-based pruning instead of other-work similarity pruning | Llama 3 · MMLU 5-shot | accuracy: 53.44 -> 53.44 | No clear effect |
| Use zigzag-based pruning instead of other-work similarity pruning | Llama 3 · HellaSwag 5-shot | accuracy: 41.60 -> 41.60 | No clear effect |
| Use zigzag-based pruning instead of other-work similarity pruning | Llama 3 · WinoGrande 5-shot | accuracy: 70.00 -> 70.00 | No clear effect |
| Use zigzag-based pruning instead of other-work similarity pruning | Mistral 7B · MMLU 5-shot | accuracy: 38.20 -> 53.17 | Helped |
| Use zigzag-based pruning instead of other-work similarity pruning | Mistral 7B · HellaSwag 5-shot | accuracy: 34.45 -> 36.67 | Helped |
| Use zigzag-based pruning instead of other-work similarity pruning | Mistral 7B · WinoGrande 5-shot | accuracy: 63.76 -> 66.50 | Helped |
| Use zigzag-based pruning instead of other-work similarity pruning | Pythia 6.9B · HellaSwag 5-shot | accuracy: 34.96 -> 31.43 | Hurt |
| Use zigzag-based pruning instead of other-work similarity pruning | Pythia 6.9B · WinoGrande 5-shot | accuracy: 58.09 -> 55.71 | Hurt |
Surprising findings
- Overall model performance is inversely related to the peak height of inter-layer persistence of long-lived features across the four models.
- Layer pruning by the zigzag criterion often selects different layers than the baselines yet yields comparable accuracy.
Limitations
- Results depend on hyperparameter choices (kNN size k, max simplex dimension) that may not be optimal; the kNN graph gives connected components with little information without the VR modification.
- Only the last token representation is analyzed as a proxy for the full prompt; analysis limited to static pretrained models, not training dynamics.
- Pruning gains are only comparable to, not clearly better than, state-of-the-art baselines, and results are mixed across benchmarks and models.
Experimental setup
Datasets: Stanford Sentiment Treebank (SST) Pile (pile-10k subset) mathematical problems dataset GitHub code (codeparrot/github-code)
Benchmarks: MMLU HellaSwag WinoGrande CMMLU CommonsenseQA WSC
Metrics: accuracy births' relative frequency inter-layer persistence
Baselines: angular distance layer pruning Block-Influence (Bi-score) layer pruning
Builds on: zigzag persistence fastzigzag persistent homology ShortGPT block-influence layer pruning angular distance layer pruning
Closest papers by meaning
- Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.86)
- The Same but Different: Structural Similarities and Differences in Multilingual Language Modeling (ICLR 2025 · similarity 0.85)
- Correlation Dimension of Autoregressive Large Language Models (NeurIPS 2025 · similarity 0.85)
- Geometry of Decision Making in Language Models (NeurIPS 2025 · similarity 0.85)
- Towards Scalable Topological Regularizers (ICLR 2025 · similarity 0.85)
- Quantifying Generalization Complexity for Large Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.