Research map / Large language models / Reasoning and chain of thought

Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MERCI, a count-based intrinsic reward for GRPO/DAPO in LLM reasoning, using Coin Flipping Network pseudo-counts and a simplified Uncertainty Bellman Equation, improving math and SQL generation results.

Problem

Sparse outcome rewards and limited exploration in RL for LLM reasoning cause repetitive, suboptimal reasoning patterns; existing epistemic-uncertainty and pseudo-count methods do not scale to LLMs.

Approach

Because LLM transitions are deterministic, the Uncertainty Bellman Equation reduces to accumulating local reward variance along a trajectory; a Coin Flipping Network estimates per-token pseudo-counts, and the square root of cumulative variance, filtered and group-normalized, is added as a clipped advantage bonus to GRPO/DAPO.

Key results

• On Qwen2.5-Math-7B, DAPO w/ MERCI reaches 69.0 average pass@k vs 66.9 for DAPO, and 44.9 vs 42.2 average mean@k.
• On Llama-3.1-8B-Instruct SQL, GRPO w/ MERCI improves Spider greedy accuracy from 74.7 to 78.0 and Bird greedy from 60.7 to 63.0.
• Ablations show removing noise filtering drops GRPO+MERCI mean@k average from 42.2 to 38.9, below the 40.5 GRPO baseline.

Surprising findings

• Extended vanilla GRPO training (to step 260) increases mean@k but degrades pass@k, while MERCI improves both.
• Without noise filtering, MERCI underperforms vanilla GRPO in mean@k, indicating instability from unfiltered bonuses.

Limitations

• Gains on some benchmarks are small or mixed (e.g., Minerva and MATH500 pass@k slightly lower than GRPO), and single-seed differences are small.
• Requires several heuristic filtering stages (percentile, spatial coherence, noise suppression) and hyperparameters such as gamma and alpha.
• Evaluated only on math and SQL with two base models; the deterministic-transition argument is tied to self-contained tasks.

Experimental setup

Datasets: DAPO-17K BIRD train

Benchmarks: AIME2024 AIME2025 MATH500 OlympiadBench College Math Minerva BIRD Spider

Metrics: pass@k mean@k greedy accuracy

Baselines: GRPO DAPO

Builds on: GRPO DAPO Coin Flipping Network (CFN) Uncertainty Bellman Equation veRL

Closest papers by meaning

• Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.91)
• Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models (NeurIPS 2026 · similarity 0.91)
• Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.90)
• Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search (ICML 2025 · similarity 0.90)
• A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.90)
• Boosting LLM Reasoning via Human-Inspired Reward Shaping (NeurIPS 2026 · similarity 0.90)