Research map / Architectures and efficiency / Attention and transformers

Augmenting Transformers with Recursively Composed Multi-grained Representations

ICLR 2024 · Architectures and efficiency › Attention and transformers · approach: Efficient language transformers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ReCAT, a Transformer augmented with stacked contextual inside-outside (CIO) layers that induce hierarchical syntax from raw text and produce multi-grained, contextualized span representations, jointly pre-trained with MLM on WikiText-103.

Problem

Transformers represent syntax and span-level structure implicitly, while explicit recursive models (RvNNs, DIORA, R2D2) restrict information flow to inside or outside a span, preventing cross-span communication and joint pre-training with Transformers.

Approach

Contextual inside-outside layers perform a pruned inside pass and a contextual outside pass that mixes inside and outside information, stacked iteratively and inserted before Transformer layers so that span representations are fully contextualized and jointly pre-trainable with MLM, while an unsupervised parser induces trees.

Key results

• ReCAT[3,1,3] reaches 92.7 SRL and 98.5 CTL F1 on OntoNotes dev, outperforming similarly sized Transformers by around 4 F1 on average.
• ReCAT noshare[3,1,3] achieves 65.00 unsupervised parsing F1 on PTB, above Fast-R2D2 (57.22) and comparable to TN-PCFG (64.1).
• ReCAT noshare[3,1,6] reaches 76.33 MNLI-m, while ReCAT noshare[3,1,3] outperforms Transformer x6 on MNLI-m/mm in the ablation discussion of the paper.

Measured effects reported in the paper

Surprising findings

• ReCAT underperforms Transformer x9 on single-sentence tasks such as CoLA and SST-2, which the authors attribute to losing local-span expressiveness in exchange for contextualization.
• Parser+Transformer, which uses gold PTB trees, is worse than Fast-R2D2+Transformer, especially on CoLA.

Limitations

• Substantial computational load during training and inference compared with vanilla Transformers.
• Underperforms vanilla Transformers on single-sentence GLUE tasks such as CoLA and SST-2.
• Gap between shared and non-shared compose functions is not significant on downstream tasks.

Experimental setup

Datasets: WikiText-103 OntoNotes 5.0 GLUE Penn Treebank

Benchmarks: OntoNotes 5.0 span-level tasks (NEL, SRL, CTL, COREF) GLUE Penn Treebank grammar induction

Metrics: F1 accuracy unsupervised parsing F1 constituent recall

Baselines: Transformer Fast-R2D2 DIORA*+Transformer Parser+Transformer BERT GumbelTree CRvNN Ordered Memory C-PCFG NBL-PCFG TN-PCFG S-DIORA

Builds on: DIORA R2D2 Fast-R2D2 Transformer deep inside-outside algorithm

Closest papers by meaning

• Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025 · similarity 0.87)
• Banyan: Improved Representation Learning with Explicit Structure (ICML 2025 · similarity 0.86)
• ReFT: Representation Finetuning for Language Models (NeurIPS 2024 · similarity 0.86)
• Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.85)
• How Transformers Learn Structured Data: Insights From Hierarchical Filtering (ICML 2025 · similarity 0.85)
• Learning and Transferring Sparse Contextual Bigrams with Linear Transformers (NeurIPS 2024 · similarity 0.85)