Research map / Architectures and efficiency / RNNs and normalization
Banyan: Improved Representation Learning with Explicit Structure
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Banyan introduces entangled hierarchical trees and diagonalised message passing in a self-structuring autoencoder, learning sentence/word embeddings from ~10M tokens with only 14 non-embedding parameters and outperforming Self-StrAE and SimCSE-RoBERTa on English STS and beating several transformers on low-resource language semantic relatedness.
Problem
Efficient learning of semantic representations in low-resource settings, where transformers need large data and compute and prior structured models like Self-StrAE underperform.
Approach
Entangle the induced binary merge trees across a batch into a single graph with shared constituent nodes (global frontier), so downward embeddings aggregate contexts, and replace linear composition/decomposition layers with sigmoid-gated diagonal functions, enabling cross-entropy reconstruction instead of a contrastive objective.
Key results
- On English sentence-level STS, Banyan reaches a score of 62.97 vs 59.02 for RoBERTa+SimCSE and 46.59 for Self-StrAE, with only 14 non-embedding parameters.
- Banyan achieves Quora NDCG@10 of 65.71 vs 59.30 for RoBERTa+SimCSE, and Arguana NDCG@10 of 28.28 vs 21.84.
- On the multilingual evaluation, Banyan averages 60.01 score, above XLM-R fine-tuned (55.84) and Paraphrase XLM-R (59.16), with strongest gains on low-resource languages such as Hausa (49.68).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Self-StrAE with Banyan's entangled-tree model and diagonal message passing | English sentence representations · English sentence-level STS (Score) | Spearman's ρ × 100: 46.59 ± 0.43 -> 62.97 ± 0.03 | Helped |
| use Banyan instead of RoBERTa + SimCSE | English sentence representations · English sentence-level STS (Score) | Spearman's ρ × 100: 59.02 ± 1.45 -> 62.97 ± 0.03 | Helped |
| use Banyan instead of Sent2Vec | English word representations · English word-level semantic similarity (Score) | Spearman's ρ × 100: 50.56 ± 0.79 -> 48.53 ± 1.33 | Hurt |
| use Banyan instead of RoBERTa + SimCSE | English retrieval representations · Quora | NDCG@1: 51.79 ± 2.12 -> 57.74 ± 0.10 | Helped |
| use Banyan instead of Sent2Vec | English frozen representations with an MLP classifier · SST-2 | accuracy: 76.53 ± 0.98 -> 75.96 ± 0.57 | Hurt |
| use Banyan instead of GloVe | English frozen representations with an MLP classifier · MRPC | F1: 81 ± 0 -> 79.48 ± 0.42 | Hurt |
| use Banyan instead of Paraphrase XLM-R | Multilingual representations · Multilingual semantic relatedness (Score) | score: 59.16 -> 60.01 | Helped |
| add diagonal composition and decomposition functions (ablation) | Standard Trees · English sentence-level STS (Score) | Spearman's ρ × 100: 46.59 ± 0.43 -> 49.68 ± 0.06 | Helped |
| replace the contrastive objective with cross-entropy loss (ablation) | Standard Trees + diagonal functions · English sentence-level STS (Score) | Spearman's ρ × 100: 49.68 ± 0.06 -> 59.91 ± 0.54 | Helped |
| use entangled trees instead of standard trees (ablation) | Self-StrAE with its original functions and objective · English sentence-level STS (Score) | Spearman's ρ × 100: 46.59 ± 0.43 -> 51.31 ± 0.13 | Helped |
| add diagonal composition and decomposition functions (ablation) | Entangled Trees · English sentence-level STS (Score) | Spearman's ρ × 100: 51.31 ± 0.13 -> 55.23 ± 0.01 | Helped |
| replace the contrastive objective with cross-entropy loss (ablation) | Entangled Trees + diagonal functions · English sentence-level STS (Score) | Spearman's ρ × 100: 55.23 ± 0.01 -> 62.97 ± 0.03 | Helped |
Surprising findings
- Cross-entropy reconstruction outperforms the contrastive objective once diagonal message passing is used, reversing the earlier finding that contrastive loss was critical.
- Banyan performs worse than transformer baselines on high-resource Spanish, and is weak on SimLex (14.65).
Limitations
- Structure is still induced by greedy agglomerative clustering, which the authors call sub-optimal.
- Banyan underperforms transformers on high-resource languages such as Spanish and on SimLex word similarity.
- Evaluation is limited to small-scale pretraining (about 10M tokens) and unsupervised probes plus frozen-representation classifiers.
Experimental setup
Datasets: English Wikipedia subsample Wiki-103 Leipzig Corpora Collection Amharic corpus (HuggingFace) OPUS (Hausa) STS-12 STS-13 STS-14 STS-15 STS-16 STS-B SICK-R
Benchmarks: STS-12 STS-13 STS-14 STS-15 STS-16 STS-B SICK-R SemRel Simlex-999 WordSim-S WordSim-R Quora
Metrics: Spearman correlation NDCG@1 NDCG@10 recall@1 recall@10 accuracy F1
Baselines: Self-StrAE GloVe Sent2Vec RoBERTa RoBERTa + SimCSE XLM-R Llama-3.1 8B Mistral Nemo MiniLM-L12 Paraphrase XLM-R
Builds on: Self-StrAE SimCSE GloVe Sent2Vec
Closest papers by meaning
- How Transformers Learn Structured Data: Insights From Hierarchical Filtering (ICML 2025 · similarity 0.88)
- Language Models as Hierarchy Encoders (NeurIPS 2024 · similarity 0.87)
- Layer by Layer: Uncovering Hidden Representations in Language Models (ICML 2025 · similarity 0.86)
- Augmenting Transformers with Recursively Composed Multi-grained Representations (ICLR 2024 · similarity 0.86)
- Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models (NeurIPS 2025 · similarity 0.86)
- Controllable Data Generation with Hierarchical Neural Representations (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.