Research map / Large language models / Alignment and preferences
Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
Shokichi Takakura, Akifumi Wachi, Rei Higuchi, Kohei Miyaguchi, Taiji Suzuki
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| meta-train with non-linear GRPO (IAMA) instead of standard GRPO on the averaged reward | Mistral 7B Instruct · Length reward on UltraFeedback (target lengths 50 and 150) | response length distribution of base model | Improved |
| increase N in BoN (or lower tau in soft BoN) in the IAMA objective (ablation) | Mistral 7B Instruct with non-linear GRPO · Length reward on UltraFeedback | polarization of response lengths | Improved |
| apply BoN (N=4) sampling at inference on IAMA-trained model | Alpaca 7B with non-linear GRPO · hh-rlhf helpfulness vs harmlessness | Pareto front of golden rewards | Improved |
| use IAMA instead of standard alignment when no BoN sampling is applied at inference | Alpaca 7B · hh-rlhf helpfulness vs harmlessness | golden rewards without BoN | No clear effect |
Closest papers by meaning
- MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models (NeurIPS 2024 · similarity 0.91)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.91)
- Alignment of Large Language Models with Constrained Learning (NeurIPS 2025 · similarity 0.91)
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.90)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.90)
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
