Research map / Reinforcement learning / RL from feedback
Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision
Dulhan Jayalath, Shashwat Goel, Thomas Foster, Parag Jain, Suchin Gururangan, Cheng Zhang, Anirudh Goyal, Alan Schelten
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with CaT (RL on synthesized pseudo-reference) | Llama 3.1 8B · HealthBench | rubric score: 0.29 -> 0.38 | Improved |
| train with CaT | Llama 3.1 8B · MATH-500 | accuracy: 0.42 -> 0.56 | Improved |
| train with CaT instead of using inference-time synthesis | Qwen 3 4B · HealthBench | rubric score: 0.42 -> 0.43 | Improved |
| train with CaT instead of using inference-time synthesis | Qwen 3 4B · MATH-500 | accuracy: 0.81 -> 0.83 | Improved |
| use RL with rubric rewards instead of SFT on pseudo-references (ablation) | CaT, Llama 3.1 8B · HealthBench | rubric score: 0.28 -> 0.38 | Improved |
| use self-proposed rubrics instead of model-as-judge reward (ablation) | CaT, Llama 3.1 8B · HealthBench | rubric score: 0.18 -> 0.38 | Improved |
| use physician rubrics instead of self-proposed rubrics (ablation) | CaT, Llama 3.1 8B · HealthBench | rubric score: 0.38 -> 0.40 | No clear effect |
| condition rubric generation on pseudo-reference (ablation) | CaT, Llama 3.1 8B · HealthBench | rubric score: 0.34 -> 0.38 | Improved |
| use synthesis instead of Self-BoN for reference estimation | Qwen 3 4B · HealthBench (inference) | rubric score: 0.40 -> 0.44 | Improved |
| use synthesis instead of majority vote | Qwen 3 4B · MATH-500 (inference) | accuracy: 0.81 -> 0.86 | Improved |
| use synthesis instead of Self-BoN | Llama 3.1 8B · MATH-500 (inference) | accuracy: 0.54 -> 0.53 | No clear effect |
| condition synthesis on 8 rollouts instead of 1 (ablation) | synthesis, Qwen 3 4B · MATH-500 | accuracy: 0.79 -> 0.85 | Improved |
Closest papers by meaning
- NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning (ICLR 2026 · similarity 0.86)
- CoRe: Collaborative Reasoning via Cross Teaching (ICML 2026 · similarity 0.85)
- Your Teacher Can’t Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation (NeurIPS 2026 · similarity 0.85)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.85)
- Let's (not) just put things in Context: Test-time Training for Long-context LLMs (ICLR 2026 · similarity 0.85)
- Reasoning with Sampling: Your Base Model is Smarter Than You Think (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
