Research map / Label-efficient and robust learning / Knowledge distillation
Regret Pre-training: Bridging Prior and Posterior Views for Enhanced Knowledge Grounding
Mingkuan Zhao, Xiayu Sun, Wentao Hu, Suquan Chen, Jiaxuan Li, Xiaoyan Zhu, Xin Lai, Jiayin Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add global future-conditioned teacher regret loss (alpha=1.0) | OLMoE-1B-7B causal LM · BoolQ (zero-shot) | accuracy (%): 42.9 -> 61.0 | Improved |
| Add local one-token-lookahead teacher regret loss | OLMoE-1B-7B causal LM · BoolQ (zero-shot) | accuracy (%): 42.9 -> 48.5 | Improved |
| Add global regret loss | OLMoE-1B-7B causal LM · 7-task average | accuracy (%): 30.2 -> 33.9 | Improved |
| Add local regret loss | OLMoE-1B-7B causal LM · 7-task average | accuracy (%): 30.2 -> 32.2 | Improved |
| Use global teacher instead of local teacher (ablation) | Regret Pre-training · ARC-Challenge | accuracy (%): 23.4 -> 25.8 | Improved |
| Use local teacher instead of global teacher (ablation) | Regret Pre-training · CommonsenseQA | accuracy (%): 23.4 -> 26.5 | Improved |
| Use local teacher instead of global teacher (ablation) | Regret Pre-training · HellaSwag | accuracy (%): 25.1 -> 27.5 | Improved |
| Use local teacher instead of global teacher (ablation) | Regret Pre-training · PIQA | accuracy (%): 57.2 -> 58.9 | Improved |
| Use local vs global teacher (ablation) | Regret Pre-training · WinoGrande | accuracy (%): 50.6 vs 50.8 | No clear effect |
| Set regret weight alpha=0.1 instead of 1.0 (ablation) | GlobalRegret · downstream tasks | accuracy improvement over baseline | No clear effect |
| Increase regret weight alpha to 2.0 (ablation) | GlobalRegret · downstream tasks | accuracy improvement over baseline | Hurt |
| Use alpha=1.0 regret weight (ablation) | GlobalRegret · downstream tasks | median improvement over baseline | Improved |
Closest papers by meaning
- Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (NeurIPS 2026 · similarity 0.87)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.86)
- Steering Information Utility in Key-Value Memory for Language Model Post-Training (NeurIPS 2025 · similarity 0.85)
- MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.85)
- Towards Auto-Regressive Next-Token Prediction: In-context Learning Emerges from Generalization (ICLR 2025 · similarity 0.85)
- Chain of Hindsight aligns Language Models with Feedback (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
