Research map / Large language models / Retrieval and knowledge
Enhancing LLM’s Cognition via Structurization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes context structurization, which reorganizes plain long text into a three-layer scope/aspect/description hierarchy, distills this ability from Qwen-Max into a 7B StruXGPT model, and shows gains across QA, hallucination evaluation, and dense retrieval.
Problem
LLMs process long, unordered contexts sequentially, which can limit their handling of intricate inputs; the paper targets improving LLM cognition over long-form text without altering the models.
Approach
Single-round context structurization rewrites the input into a well-ordered three-layer hierarchy (scope, aspects, descriptions) with linguistic markers, and a 7B model is fine-tuned via SFT on teacher outputs to perform this structurization, so the downstream LLM is left unchanged.
Key results
- Structurization raises LLaMA2-70B hallucination evaluation accuracy by 4.3% on average on AttrScore, and GPT-3.5 from 58.0% to 62.1%.
- On LongBench, LLaMA2-13B-4k average rises from 22.9 to 26.0 with StruXGPT-7B structurized inputs.
- StruXGPT-7B reaches about 97% of the teacher's structurization ability per the paper's human evaluation and AppEval comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Structurize long-form contexts before QA | LLaMA2-7B-4k, LLaMA2-13B-4k, Qwen-7B-8k, and ChatGLM3-6B-32k · LongBench | Average: 19.1 -> 21.4; 22.9 -> 26.0; 15.9 -> 18.2; 47.9 -> 49.3 | Helped |
| Structurize passages while leaving queries unchanged | BERT, SimLM, and coCondenser · BEIR | Average nDCG@10: 29.3 -> 30.3; 26.7 -> 28.3; 32.0 -> 33.1 | Helped |
| Structurize reference passages | Qwen-7B and Qwen-72B evaluators · AttrScore and FactScore | Accuracy: AttrScore: 60.9 -> 61.9, 11.6 -> 25.9, 34.2 -> 44.4; FactScore: 68.5 -> 79.2, 50.8 -> 64.8, 40.8 -> 46.0 | Helped |
| Add structurization to GPT-3.5-Turbo | GPT-3.5-Turbo · AttrScore | Average accuracy: 58.0 -> 62.1 | Helped |
| Add structurization to GPT-3.5-Turbo with CoT | GPT-3.5-Turbo + CoT · AttrScore | Average accuracy: 62.0 -> 65.4 | Helped |
| Add structurization to GPT-4 with CoT | GPT-4 + CoT · AttrScore | Average accuracy: 75.8 -> 77.2 | Helped |
| Use 1 rather than 2 few-shot examples to generate teacher data (ablation) | Teacher prompting for StruXGPT training data · Qasper | AppEval improvement: +3.2 -> +1.8 | Hurt |
| Use 3 rather than 2 few-shot examples to generate teacher data (ablation) | Teacher prompting for StruXGPT training data · Qasper | AppEval improvement: +3.2 -> +3.3 | No clear effect |
| Increase StruXGPT capacity from 1.8B to 7B (ablation) | Qwen-based StruXGPT · Qasper | AppEval improvement: +2.7 -> +3.6 | Helped |
| Increase StruXGPT capacity from 7B to 14B (ablation) | Qwen-based StruXGPT · Qasper | AppEval improvement: +3.6 -> +3.8 | No clear effect |
| Filter training pairs with negative BERTScore (ablation) | StruXGPT-7B · Qasper | AppEval improvement: +3.6 -> +3.4 | Hurt |
| Filter low-BERTScore inference structurizations and fall back to original context (ablation) | LLaMA2-7B-Chat · Qasper | Overall enhancement: +3.6 -> +3.7 | Helped |
Surprising findings
- Structurization slightly lowers MMLU (45.93 to 45.84) for LLaMA2-7B-Chat, indicating it does not help context-free knowledge tasks.
- Pre-trained 7B models with few-shot prompting fail to produce valid structurizations, while the fine-tuned StruXGPT-7B does.
Limitations
- Inference-time overhead from an extra structurization pass adds noticeable latency, especially for small evaluators.
- Gains are negligible or negative on some tasks, such as Musique for ChatGLM3 and context-free MMLU.
- Only a fixed three-layer structure is studied; richer structures such as mindmaps are left for future work.
Experimental setup
Datasets: Wikipedia 20231020-en dump CAMEL-AI FiQA MedQuad LongBench AttrScore AttrEval-GenSearch FactScore BEIR MS MARCO
Benchmarks: LongBench (Qasper, MultiFieldQA, HotpotQA, 2WikiMQA, Musique, PassageCount, PassageRetrieval) AttrScore FactScore BEIR (NFCorpus, FiQA, ArguAna, SciDocs, SciFact) MMLU BBH
Metrics: F1 ROUGE-L accuracy nDCG@10 human score 0-5 BERTScore ROUGE-L recall/precision
Baselines: LLaMA2-7B-4k LLaMA2-13B-4k Qwen-7B-8k ChatGLM3-6B-32k GPT-3.5-Turbo GPT-4 AttrScore models (Alpaca) ABS QBS AdaptLLM BERT SimLM
Builds on: chain-of-thought aspect-based summarization query-based summarization LongBench
Closest papers by meaning
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.89)
- Mixture of In-Context Experts Enhance LLMs' Long Context Awareness (NeurIPS 2024 · similarity 0.89)
- Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension ability (ICLR 2025 · similarity 0.89)
- Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs (ICLR 2024 · similarity 0.88)
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.88)
- Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.