Research map / Large language models / Language, safety and interpretability
EntropyLong: Effective Long-Context Training via Predictive Uncertainty
Junlong Jia, Ziyang Chen, Xing W, Chaochen Gao, Zijia Lin, Songlin Hu, GuoBinghui
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| entropy-verified long-range dependency data (EntropyLong) vs NExtLong | Llama-3-8B · RULER avg | score: 85.22 -> 87.37 | Improved |
| EntropyLong vs NExtLong at 128k | Llama-3-8B · RULER 128k | score: 77.99 -> 81.26 | Improved |
| EntropyLong data then SFT | Llama-3-8B · LongBench-v2 overall | accuracy: 24.10 -> 27.60 | Improved |
| EntropyLong data then SFT | Llama-3-8B · LongBench-v2 Long | accuracy: 23.10 -> 31.50 | Improved |
| add empirical entropy-reduction verification (ablation) | EntropyLong · RULER avg | score: 85.82 -> 87.37 | Improved |
| use alpha=2.0 instead of 1.5 (ablation) | EntropyLong · RULER avg | score: 82.49 -> 87.37 | Improved |
| use alpha=2.0 instead of 2.5 (ablation) | EntropyLong · RULER avg | score: 85.52 -> 87.37 | Improved |
| use verification threshold 0.4 instead of 0.2 (ablation) | EntropyLong · RULER avg | score: 85.45 -> 87.37 | Improved |
| use verification threshold 0.4 instead of 0.6 or 0.8 (ablation) | EntropyLong · RULER avg | score: 86.14 / 86.47 -> 87.37 | Improved |
| window size 16 instead of 2, 4, 8 (ablation) | EntropyLong · RULER avg | score: 86.04-86.85 -> 87.37 | Mixed |
| random shuffle instead of sequence concatenation (ablation) | EntropyLong · RULER avg | score: 87.06 -> 87.37 | No clear effect |
| train with EntropyLong data | Llama-3-8B · short-text benchmarks avg | accuracy: 61.77 -> 61.83 | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- NExtLong — NExtLong: Toward Effective Long-Context Training without Long Documents (ICML 2025) · better on 5 of 5 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- NExtLong: Toward Effective Long-Context Training without Long Documents (ICML 2025 · similarity 0.90)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.89)
- Structured Packing in LLM Training Improves Long Context Utilization (AAAI 2025 · similarity 0.88)
- Long-Short Alignment for Effective Long-Context Modeling in LLMs (ICML 2025 · similarity 0.88)
- L$^2$M: Mutual Information Scaling Law for Long-Context Language Modeling (NeurIPS 2025 · similarity 0.88)
- Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
