Research map / Reinforcement learning / Visual and representation RL

Enhancing Question Generation through Diversity-Seeking Reinforcement Learning with Bilevel Policy Decomposition

AAAI 2025 · Reinforcement learning › Visual and representation RL · approach: Hierarchical skills and priors · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BPD-DSRL, which decomposes question generation into a rationale (answer-centric summary) policy and a question policy, trained with a diversity-seeking GFlowNet-style trajectory-balance objective against NLI and fluency/answerability reward models.

Problem

Reinforcement learning for question generation (RLQG) suffers from poor sample efficiency because the source-target mapping is implicit, and from loss of diversity because reward maximization collapses policies onto a single high-return question.

Approach

A bilevel policy decomposition splits QG into answer-centric summary generation and summary-augmented question generation, and a diversity-seeking RL objective (trajectory-balance style, log-scale L2) trains the policies to sample proportionally to reward rather than maximize return.

Key results

• BPD-DSRL achieves Top-1 BLEU-4 of 19.92 on SQuAD 1.1/1, 19.66 on SQuAD 1.1/2, and 14.13 on NewsQA, the best among non-RAG methods, and beats RAST on Top-1 across datasets while lowering Self-BLEU versus other RLQG baselines.
• On SQuAD 1.1/1, BPD-DSRL gains 2.61% Top-1 and 2.58% Oracle over its SFT start, versus 0.21% and 0.72% for DSRL without BPD.
• BPD-DSRL reaches Self-BLEU 55.28 on SQuAD 1.1/1 versus 56.85 for BPD-PPO and 67.79 for BPD-REINFORCE, indicating greater diversity among RLQG methods.

Measured effects reported in the paper

Surprising findings

• BPD-REINFORCE produces the most extractive questions and highest GPT-3.5 answer accuracy, which the authors attribute to possible over-optimization.
• RAST, a retrieval-augmented CEQG method, has the best Self-BLEU diversity on several splits, and the authors exclude RAG-based methods from the diversity comparison.

Limitations

• Diversity cost of rationale conditioning is acceptable only on SQuAD; on NewsQA the diversity trade-off is less favorable for some variants.
• Integration with retrieval-augmented RAST is deferred to future work, so the best diversity from RAG-based methods is not matched.
• Evaluation relies mainly on BLEU-based metrics, with a smaller set of GPT-3.5 and human checks.

Experimental setup

Datasets: SQuAD 1.1 (Zhou et al. 2017 split) SQuAD 1.1 (Du et al. 2017 split) NewsQA

Benchmarks: SQuAD 1.1 / 1 SQuAD 1.1 / 2 NewsQA

Metrics: Top-1 BLEU-4 Oracle BLEU-4 Self-BLEU hallucination rate (spaCy NER) answerability rate answer accuracy Self-BERTScore

Baselines: Composition RAST LLaMA3-8B-Instruct (0-shot, 3-shot) Mistral-7B-Instruct-v0.3 (0-shot, 3-shot) BPD-REINFORCE BPD-PPO

Builds on: Trajectory Balance (GFlowNets) Soft Q-learning REINFORCE PPO Composition RAST

Closest papers by meaning

• Diversity-Enhanced Reasoning for Subjective Questions (ICLR 2026 · similarity 0.86)
• Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
• Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity (ICLR 2026 · similarity 0.86)
• SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning (ICML 2026 · similarity 0.85)
• Jointly Reinforcing Diversity and Quality in Language Model Generations (NeurIPS 2026 · similarity 0.85)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.84)