Research map / Architectures and efficiency / Attention and transformers
AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdmTree, a hierarchical context compressor that adaptively segments input by information density, builds a binary semantic tree over gist-token leaves, and trains only lightweight attention and aggregation modules on a frozen LLM.
Problem
Long-context processing with LLMs is costly due to quadratic self-attention, and existing context compression methods lose local detail (explicit), suffer positional bias or degradation over recursive compression (implicit).
Approach
Adaptive variable-length segmentation with gist-token budgets allocated by an entropy-adjusted perplexity score, aggregated bottom-up into a binary semantic tree via a single-layer self-attention aggregator, with separate trainable projections for gist tokens and a frozen backbone for text tokens.
Key results
- On LongBench with LLaMA-2-7B, AdmTree averages 44.1 vs. 40.1 for Activation Beacon (about 10% gain) and 49.7 vs. 47.2 with Qwen-2-7B.
- On LongBench v2, AdmTree reaches 23.1 AVG with LLaMA-2-7B vs. 19.5 for Activation Beacon.
- On GSM8K (quarter-state constant, ~20x compression), AdmTree scores 46.85 EM, above the full-shot baseline of 46.88 only nearly; on BBH it reaches 53.53 at ~7x compression vs. 43.34 full-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AdmTree instead of Beacon | LongBench with LLaMA-2-7B · LongBench | AVG | Helped |
| Use AdmTree instead of Beacon | LongBench with Qwen-2-7B · LongBench | AVG | Helped |
| Use AdmTree instead of Beacon | LongBench v2 with LLaMA-2-7B · LongBench v2 | AVG: +18.5% | Helped |
| Use AdmTree instead of Beacon | LongBench v2 with Qwen-2-7B · LongBench v2 | AVG: +12.7% | Helped |
| Increase compression from 1-st. const. to quarter-st. const. | AdmTree on BBH · BBH | Exact Match: 45.06 -> 53.53 | Helped |
| Add pre-training (ablation) | AdmTree · Compression learning | performance | Helped |
| Add instruction fine-tuning (ablation) | AdmTree · Compression learning | performance | Helped |
| Add the semantic tree structure (ablation) | AdmTree · Compression benchmark | performance | Helped |
| Use adaptive rather than uniform leaf gist token allocation (ablation) | AdmTree · Compression benchmark | performance | Helped |
| Add self-attention parameters to tree aggregation (ablation) | AdmTree · Compression benchmark | performance | Helped |
| Retrieve only the top 75% of tree nodes during inference (ablation) | AdmTree · Tree-node retrieval | performance: 75% | Hurt |
Surprising findings
- Activation Beacon shows hallucinated or looping output in the repetition case study, while AdmTree reconstructs the text near-perfectly.
- On BBH, higher compression ratios yielded better AdmTree performance, which the authors attribute to short full-shot prompts.
Limitations
- Results are reported for only two 7B backbones and fixed compression ratio schedules tied to maximum sequence length.
- Code and model release is promised but not yet available at submission; the paper itself notes MoE specialization as only a potential direction.
- Reported gains on some tasks depend on the specific baselines and training data matched to Activation Beacon.
Experimental setup
Datasets: RedPajama LongAlpaca BookSum GPT-3.5 synthetic samples LongBench LongBench v2 Needle-in-a-Haystack GSM8K BBH ShareGPT MSC ArXiv-March23
Benchmarks: LongBench LongBench v2 Needle-in-a-Haystack GSM8K BBH ShareGPT MSC ArXiv-March23 Topic Retrieval
Metrics: accuracy F1 ROUGE BERTScore BLEU PPL latency TFLOPs exact match
Baselines: BM25 Sentence-BERT OpenAI Embeddings LongLLMLingua AutoCompressor ICAE Activation Beacon SnapKV Original LLM-FT
Builds on: Activation Beacon AutoCompressor ICAE Gisting LongLLMLingua
Closest papers by meaning
- From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMs (ICLR 2025 · similarity 0.88)
- Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs (ICLR 2024 · similarity 0.87)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.87)
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.86)
- Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search (NeurIPS 2025 · similarity 0.86)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.