Research map / Reinforcement learning / RL from feedback

ExpLang: Improved Exploration and Exploitation in LLM Reasoning with On-Policy Thinking Language Selection

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ExpLang, an RLVR pipeline where a Qwen3-4B model learns on-policy thinking-language selection (LoRA SFT cold start, then exploration/exploitation GRPO stages with language diversity and language Pass@k rewards), improving math reasoning over English-only training.

Problem

English-centric RL post-training of large reasoning models ignores potential benefits of multilingual thinking for exploration and exploitation, and native-language thinking traces are needed by global users.

Approach

Equip the model with a thinking-language selection tag learned via LoRA SFT, then let it choose its thinking language on-policy during RLVR, using a language-diversity reward during an exploration stage and a language-grouped Pass@k reward during an exploitation stage.

Key results

• On MATH-500, RLVR (Ours) reaches 91.5% accuracy vs 91.4% for naive GRPO and 91.0% for controlled baseline under automatic selection, with fewer tokens (1263 vs 1417).
• On AIME-2025 with automatic selection, ExpLang reaches 31.9% (+15.8 over SFT start) vs 30.8% naive GRPO and 26.4% controlled baseline.
• On unseen languages (MATH-500), ExpLang's Acc* is 61.5% vs 24.6% naive and 37.9% controlled, with 75.5% compliance.

Surprising findings

• Naive English-only GRPO nearly matches ExpLang on MATH-500 automatic accuracy (91.4 vs 91.5), with the gain concentrated on AIME and unseen-language settings.
• Forced multilingual thinking shows higher Pass@k but lower accuracy than English thinking for the base Qwen3 models.

Limitations

• Experiments limited to Qwen3-4B and math reasoning with generation length capped at 4096 tokens.
• Gains over naive GRPO on MATH-500 automatic accuracy are marginal; the method is not compared with other GRPO variants in experiments.

Experimental setup

Datasets: OpenR1-Math-220k MATH-500 AIME 2025 OlymMATH en-easy

Benchmarks: MATH-500 AIME 2025 OlymMATH en-easy

Metrics: accuracy Pass@k thinking tokens thinking language compliance Acc^F Acc^*

Baselines: Qwen3-4B naive GRPO (English-only) controlled baseline (LoRA SFT on English traces + GRPO) full SFT

Builds on: GRPO RLVR Pass@k training Dr. GRPO DAPO GSPO

Closest papers by meaning

• Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs (ICLR 2026 · similarity 0.90)
• Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.90)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.89)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.89)
• Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.89)