Research map / Large language models / Language, safety and interpretability
Language Models as Hierarchy Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-trains sentence-transformer encoder LMs as Hierarchy Transformers (HiTs) by mapping their output embeddings into a Poincaré ball with a dimension-adaptive curvature and training with hyperbolic clustering and centripetal losses.
Problem
Pre-trained transformer encoder LMs poorly encode explicit hierarchical (subsumption) structure, and they do not infer transitive or unseen subsumptions; static hyperbolic embeddings lack inductive prediction ability.
Approach
Place the output embedding space of an encoder LM inside a Poincaré ball of radius sqrt(d) (curvature 1/d) that circumscribes the tanh hyper-cube, then re-train the LM with a hyperbolic triplet clustering loss plus a centripetal norm loss, adding no new learnable parameters.
Key results
- all-mpnet-base-v2+HiT reaches 0.916 F-score on WordNet Multi-hop Inference with random negatives, versus 0.339 for the pre-trained all-mpnet-base-v2.
- all-MiniLM-L12-v2+HiT achieves 0.553 F-score on WordNet-to-Schema.org transfer (random negatives), at least 0.150 above the pre-trained model.
- On WordNet-to-DOID transfer, all-MiniLM-L12-v2+HiT reaches 0.704 F-score (random negatives) versus 0.389 for the pre-trained model and 0.603 for fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply HiT hierarchy re-training instead of standard fine-tuning | all-MiniLM-L12-v2 · WordNet Mixed-hop Prediction, random negatives | F-score: 0.627 -> 0.885 | Helped |
| Apply HiT hierarchy re-training instead of standard fine-tuning | all-MiniLM-L12-v2 · WordNet Mixed-hop Prediction, hard negatives | F-score: 0.644 -> 0.871 | Helped |
| Apply HiT hierarchy re-training instead of standard fine-tuning | all-MiniLM-L12-v2 · WordNet to Schema.org Transfer, random negatives | F-score: 0.411 -> 0.553 | Helped |
| Apply HiT hierarchy re-training instead of standard fine-tuning | all-MiniLM-L12-v2 · WordNet to FoodOn Transfer, random negatives | F-score: 0.407 -> 0.554 | Helped |
| Apply HiT hierarchy re-training instead of standard fine-tuning | all-MiniLM-L12-v2 · WordNet to DOID Transfer, random negatives | F-score: 0.603 -> 0.704 | Helped |
| Train with hard rather than random negatives | all-MiniLM-L12-v2 + HiT · WordNet Mixed-hop Prediction | F-score: 0.885 -> 0.871 | Mixed |
| Use α=3.0 instead of α=5.0 (ablation) | all-MiniLM-L12-v2 + HiT · WordNet Mixed-hop Prediction | F-score: 0.885 -> 0.865 | Hurt |
| Use α=1.0 instead of α=5.0 (ablation) | all-MiniLM-L12-v2 + HiT · WordNet Mixed-hop Prediction | F-score: 0.885 -> 0.867 | Hurt |
| Use β=0.5 instead of β=0.1 (ablation) | all-MiniLM-L12-v2 + HiT · WordNet Mixed-hop Prediction | F-score: 0.885 -> 0.899 | Helped |
| Use triplet contrastive loss rather than standard contrastive or softmax contrastive loss (ablation) | HiT · Hierarchy re-training | convergence and effectiveness | Helped |
| Train on SNOMED rather than WordNet for transfer to DOID | all-MiniLM-L12-v2 + HiT · DOID Transfer Mixed-hop Prediction | F-score: 0.704 -> 0.850 | Helped |
Surprising findings
- Fine-tuned LMs achieve high precision but very low recall on transfer tasks (e.g., 0.177 recall on WordNet-to-FoodOn hard negatives), indicating poor generalisation beyond single-hop subsumptions.
- Fine-tuned all-MiniLM-L12-v2 performs even worse than its pre-trained version on Schema.org when trained on SNOMED CT, suggesting overfitting to biomedical knowledge.
Limitations
- Potential loss of pre-trained language understanding due to hierarchy re-training (catastrophic forgetting not measured).
- Entity naming ambiguity in source datasets may introduce noise into training.
- Hard-negative performance is still notably lower than random-negative performance for pre-trained LMs and some baselines.
Experimental setup
Datasets: WordNet (noun) Schema.org FoodOn DOID SNOMED CT
Benchmarks: WordNet Multi-hop Inference WordNet Mixed-hop Prediction Transfer Mixed-hop Prediction (Schema.org, FoodOn, DOID) SNOMED CT Mixed-hop Prediction
Metrics: precision recall F-score
Baselines: NaivePrior all-MiniLM-L6-v2 all-MiniLM-L12-v2 all-mpnet-base-v2 BERTSubs-style fine-tuning PoincaréEmbed HyperbolicCone PoincaréGloVe
Builds on: Poincaré embeddings hyperbolic entailment cones sentence transformers BERTSubs Geoopt
Closest papers by meaning
- HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts (NeurIPS 2025 · similarity 0.88)
- Leviathan: Decoupling Input and Output Representations in Language Models (NeurIPS 2026 · similarity 0.88)
- Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models (ICLR 2025 · similarity 0.88)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.87)
- EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.87)
- Hyperbolic Fine-Tuning for Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.