Research map / Large language models / Reasoning and chain of thought
ReLaX: Reasoning with Latent Exploration for Large Reasoning Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReLaX, an RLVR method that uses a Koopman-operator-based Dynamic Spectral Dispersion (DSD) metric on last-layer hidden states as an advantage-shaped, adaptively KL-regularized exploration bonus within GRPO, improving multimodal and text-only reasoning.
Problem
RLVR drives LRMs toward over-determinism and entropy collapse, causing ineffective exploration and premature convergence; existing fixes target token-level entropy, which may not capture internal computational rigidity especially in MLLMs.
Approach
Measure exploration via the spectral dispersion of hidden-state latent dynamics approximated by a Koopman operator (ResKoopNet dictionary frozen after first step), and regularize GRPO with a log-mean-exp penalty on DSD weighted by positive advantages, plus adaptive KL on high-DSD trajectories.
Key results
- ReLaX-VL-7B averages 53.2 across 7 multimodal benchmarks, above the prior best VL-Rethinker-7B (52.5); ReLaX-VL-3B averages 48.1.
- On Qwen2.5-7B-Math, ReLaX averages 49.1 across 6 text-only math benchmarks, vs. FR3E 42.8 and vanilla GRPO-style baselines lower.
- On Qwen2.5-7B-Base, ReLaX averages 43.5 vs. FR3E 39.2 and SimpleRL 34.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add latent-dynamics DSD regularization (ReLaX) on top of GRPO | Qwen2.5-7B-Math text-only reasoning · 6 math benchmarks average | average mean@1/mean@32: ReLaX 49.1 vs FR3E 42.8 | Helped |
| add ReLaX DSD regularization to vanilla GRPO | Qwen2.5-VL-3B multimodal · 7 multimodal benchmarks average | average mean@1 accuracy: +8.3 over base | Helped |
| add ReLaX DSD regularization to Qwen2.5-VL-7B | Qwen2.5-VL-7B multimodal · 7 multimodal benchmarks average | average mean@1 accuracy: +5.3 (47.9 to 53.2) | Helped |
| add ReLaX to Qwen2.5-7B-Base GRPO (SimpleRL) | Qwen2.5-7B-Base · 6 math benchmarks average | average mean@1/mean@32: ReLaX 43.5 vs FR3E 39.2 (+4.3) | Helped |
| add ReLaX to Qwen2.5-3B-Base GRPO | Qwen2.5-3B-Base · 6 math benchmarks average | average mean@1/mean@32: 31.6 vs 26.0 vanilla GRPO | Helped |
| add ReLaX to Qwen3-Base 4B GRPO | Qwen3-Base 4B · 6 math benchmarks average | average mean@1/mean@32: +8.2 (44.1 to 52.3) | Helped |
| add ReLaX to Llama3.2-3B-Instruct GRPO | Llama3.2-3B-Instruct · 6 math benchmarks average | average mean@1/mean@32: +5.3 (27.1 to 32.4) | Helped |
| remove adaptive KL regularization (uniform KL vs DSD-thresholded KL) (ablation) | ReLaX on Qwen2.5-7B-Math · 6 math benchmarks | average performance | Hurt |
| add advantage shaping (positive-part advantage weighting of DSD) (ablation) | ReLaX on Qwen2.5-7B-Math · 6 math benchmarks | average performance | Helped |
| increase DSD regularization coefficient alpha to 1.0 (ablation) | ReLaX on Qwen2.5-VL-3B · training reward | policy reward | Hurt |
| add ReLaX DSD regularization relative to KL-Cov on Qwen2.5-VL-3B | Qwen2.5-VL-3B multimodal · EMMA-Physics and visually grounded benchmarks | mean@1 accuracy: +7.7 on EMMA-Physics | Helped |
| add entropy regularization in multimodal training | Qwen2.5-VL-3B · multimodal benchmarks | accuracy | Hurt |
Surprising findings
- Sample-wise DSD does not strongly correlate with correctness, yet policy optimization benefits from consistently higher DSD.
- Removing advantage shaping causes training collapse and performance worse than the base model on Qwen2.5-7B-Math.
Limitations
- Adds runtime overhead from DSD computation each step (+10-12% total per step).
- Sensitive to hyperparameters: alpha, Koopman dimension m (m=5 fails), and DSD threshold xi (xi=infinity collapses).
- Koopman dictionary is fit once on initial policy hidden states and frozen, so it may not track later policy changes.
Experimental setup
Datasets: ViRL39K DAPO-Math-17K MATH training set (Level 3-5)
Benchmarks: MathVista MathVerse MathVision DynaMath MMMU MMStar EMMA MATH500 Minerva AMC22 AMC23 AIME24
Metrics: mean@1 accuracy mean@32 accuracy
Baselines: Vanilla GRPO SimpleRL DAPO KL-Cov DCPO R1-zero-Div FR3E HICRA MM-Eureka Vision-R1 R1-VL OpenVLThinker
Builds on: GRPO DAPO ResDMD ResKoopNet KL-Cov
Closest papers by meaning
- Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.90)
- Hybrid Latent Reasoning via Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Restoring Exploration after Post-Training: Latent Exploration Decoding for Large Reasoning Models (ICML 2026 · similarity 0.90)
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models (ICML 2026 · similarity 0.89)
- SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.