Research map / Reinforcement learning / RL from feedback

MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MemReward, a heterogeneous graph-based experience memory with a warmup-trained GNN that propagates rewards from 20% labeled rollouts to unlabeled rollouts during GRPO training, reaching near-Oracle performance on Qwen2.5-1.5B/3B.

Problem

RL fine-tuning of LLM reasoning needs ground-truth reward labels for every rollout, which are costly in data-scarce settings such as proof grading or open-ended QA.

Approach

Store labeled rollouts (query, thinking, answer) as nodes in a heterogeneous graph with query-query similarity, query-thinking and thinking-answer edges; train a GNN to predict rewards, then attach unlabeled rollouts by query similarity at online GRPO time and use thresholded GNN predictions as rewards.

Key results

• With 20% ground-truth labels, MemReward reaches 97.3% of Oracle average on Qwen2.5-3B (77.02 vs 79.12) and 96.6% on Qwen2.5-1.5B.
• On out-of-domain tasks, MemReward trails R1-Oracle by only 0.23 on 3B and 0.75 on 1.5B average.
• At 70% GT labels on 3B, MemReward reaches 99.4% of Oracle (78.64 vs 79.12).

Surprising findings

• MemReward on Qwen2.5-1.5B exceeds R1-Oracle on NuminaMath by 0.89 points despite using only 20% labels.
• The frozen GNN's prediction accuracy stays stable (74.9%-77.3%) across 410 GRPO steps.

Limitations

• Evaluated only on Qwen2.5 models at 1.5B and 3B scales.
• Graph construction depends on a fixed sentence encoder for query similarity.
• Code-domain GNN prediction quality is lower (ROC-AUC 0.72-0.83).

Experimental setup

Datasets: GSM8K GSM-Symbolic MATH MMLU CommonsenseQA OpenBookQA ARC-Challenge GPQA HumanEval+ MBPP+ NuminaMath SIQA

Benchmarks: GSM8K GSM-Symbolic MATH MMLU CommonsenseQA OpenBookQA ARC-Challenge GPQA HumanEval+ MBPP+ NuminaMath SIQA

Metrics: exact match accuracy pass@1 ROC-AUC

Baselines: R1-p (partial labels) R1-Oracle kNN reward prediction LLM-RM

Builds on: GRPO GATv2 HAN R-GCN

Closest papers by meaning

• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026 · similarity 0.89)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.89)
• Boosting LLM Reasoning via Human-Inspired Reward Shaping (NeurIPS 2026 · similarity 0.88)
• Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.88)
• LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards (ICLR 2026 · similarity 0.88)