Research map / Large language models / Reasoning and chain of thought

SELF-HARMONY: LEARNING TO HARMONIZE SELF-SUPERVISION AND SELF-PLAY IN TEST-TIME REINFORCEMENT LEARNING

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Self-Harmony, a label-free test-time RL framework where one model alternates as Solver and Reframer (paraphrase) and selects pseudo-labels by harmonic mean of answer frequencies across original and paraphrased views.

Problem

Test-time reinforcement learning without labels relies on pseudo-labels, but majority voting often amplifies spurious, popular wrong answers produced by systematic reasoning flaws.

Approach

A single model acts as Solver and Reframer; the pseudo-label is chosen as the answer with the highest harmonic mean of its answer frequencies across original and reframed question views, so only answers stable under paraphrase are reinforced. Reward is multiplicative (correctness gated by format and JS-divergence diversity penalties).

Key results

• On MATH500 with Qwen3-1.7B-Base, accuracy improves from 42.70% to 69.60%, versus at most 65% for baselines.
• Self-Harmony ranks first in 28 of 30 model-benchmark settings.
• Llama-3.1-8B-Instruct on GSM8K rises from 60.48% to 91.59%; Qwen3-4B-Base on MATH500 from 60.20% to 78.50%.

Surprising findings

• Removing the format reward on Qwen3-8B-Base drops MATH500 accuracy to 77.46%, below majority voting ablation (79.00%).
• Even a 0.6B Reframer yields large gains on the already-confident-correct subset, suggesting insensitivity to paraphrase quality.

Limitations

• Theoretical result relies on assumptions (view invariance, balanced confidence) that are only approximately satisfied and a second-order approximation.
• Reframed questions are not always semantically equivalent (about 88% consistency per oracle judges), and some rewrites alter answers or become trivial.
• Results are reported on base or instruct models with specific hyperparameters; hardware and compute cost are not stated.

Experimental setup

Datasets: MATH500 GSM8K AIME 2024 AMC GPQA-Diamond MMLU-Pro

Benchmarks: MATH500 GSM8K AIME 2024 AMC GPQA MMLU-Pro

Metrics: pass@1 averaged over 16 rollouts pseudo-label accuracy consistency rate

Baselines: Intuitor Rent Majority-Voting Co-Reward GT-Reward

Builds on: TTRL majority voting Infomax Co-Reward GRPO

Closest papers by meaning

• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models (ICLR 2026 · similarity 0.88)
• Meta-TTRL: A Metacognitive Framework for Self-Improving Test-Time Reinforcement Learning for T2I Generation in Unified Multimodal Models (NeurIPS 2026 · similarity 0.88)
• RESTRAIN: From Spurious Votes to Signals — Self-Training RL with Self-Penalization (ICLR 2026 · similarity 0.88)
• Self-Aligned Reward: Towards Effective and Efficient Reasoners (ICLR 2026 · similarity 0.87)
• Absolute Zero: Reinforced Self-play Reasoning with Zero Data (NeurIPS 2025 · similarity 0.87)