Research map / Large language models / Instruction and fine-tuning
HE-SNR: Uncovering Latent Logic via Entropy for Guiding Mid-Training on SWE-bench
Yueyang Wang, Jiawei Fu, Baolong Bi, Xili Wang, Xiaoqing Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compute the metric on Action tokens instead of Thinking tokens (ablation) | HE-SNR · SWE-bench Verified correlation | Pearson r: 0.5581 -> 0.9666 | Improved |
| compute the metric on Action tokens instead of Thinking tokens (ablation) | HE-SNR · SWE-bench Verified correlation | Kendall tau: 0.5192 -> 0.9440 | Improved |
| add XML tag removal to Action filtering (ablation) | HE-SNR · SWE-bench Verified correlation | Pearson r: 0.9666 -> 0.9526 | Hurt |
| add XML tag removal to Action filtering (ablation) | HE-SNR · SWE-bench Verified correlation | Kendall tau: 0.9440 -> 0.9558 | Improved |
| add AST-based comment removal (full pipeline) (ablation) | HE-SNR · SWE-bench Verified correlation | Kendall tau: 0.9676 -> 0.9794 | Improved |
| add AST-based comment removal (full pipeline) (ablation) | HE-SNR · SWE-bench Verified correlation | Pearson r: 0.9520 -> 0.9649 | Mixed |
| filter out Thought tokens and stylistic noise before computing HE-PPL (ablation) | HE-PPL · SWE-bench Verified correlation | linearity vs Pass@1 | Improved |
| use HE-SNR instead of standard PPL across Long-Context Tax | MoE-A26B mid-training · 128K extension, SWE-bench Verified | agreement with SWE-bench Pass@1 | Improved |
| vary entropy threshold epsilon over ln t, t in [2,5] (ablation) | HE-SNR · SWE-bench Verified correlation | Pearson r / Kendall tau | No clear effect |
| apply SFT to base checkpoints | MoE-A3B / MoE-A26B · high-entropy token set | HE-PPL and HE-SNR | Hurt |
| extend context window via RoPE scaling (Long-Context Tax) | MoE-A26B · 128K extension step 200 | PPL and Top-1 accuracy | Hurt |
Closest papers by meaning
- The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning (NeurIPS 2025 · similarity 0.87)
- Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy (ICLR 2026 · similarity 0.87)
- Integrating Local and Global Entropy for Uncertainty Quantification in LLMs (NeurIPS 2026 · similarity 0.87)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.86)
- Entropy-informed Decoding: Adaptive Information-Driven Branching (ICML 2026 · similarity 0.86)
- Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
