Research map / Large language models / Alignment and preferences

SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes stable rank of final-layer hidden states as an annotation-free quality reward, used zero-shot for RewardBench/Best-of-N and as a frozen-reference GRPO reward (SR-GRPO) that improves Qwen2.5-1.5B and DeepSeek-R1-Distill-Qwen-1.5B.

Problem

Aligning LLMs usually needs human preference labels or learned reward models, which are scarce, subjective, hackable, or replaced by biased self-evaluation; the paper asks whether text quality can be measured from internal representation geometry.

Approach

Use the stable rank (squared Frobenius over squared spectral norm) of final-layer hidden states from a frozen reference model as an intrinsic quality score, both as a zero-shot judge and as a dense GRPO reward with no external labels.

Key results

• Stable rank reaches 84.04% accuracy on RewardBench with Qwen3-8B, versus 83.70% for pointwise LLM scoring and 78.02% for IPO.
• SR-GRPO raises Qwen2.5-1.5B-Instruct math average from 28.0% to 32.4% and WB-Elo by 26.2 points over base.
• Best-of-N with stable-rank selection at N=16 gives +17.0% on Qwen2.5-1.5B and +20.5% on Llama-3.2-1B relative to greedy decoding.

Surprising findings

• Stable rank correlates negatively with token and sentence count, so it penalizes verbosity rather than rewarding length.
• Random best-of-N selection often degrades performance below greedy decoding on small models.

Limitations

• Correlations with text quality metrics are moderate (|rho| about 0.2-0.4).
• Evaluation limited to small models (1-1.5B) for RL, with only two policies trained.
• Stable rank accuracy drops sharply when inputs are truncated to 128 tokens (62.59% overall).

Experimental setup

Datasets: SmolTalk2 RewardBench

Benchmarks: RewardBench GPQA MMLU-redux MATH500 AIME25 OlympiadBench AMC23 WildBench

Metrics: accuracy Pass@1 WB-Elo

Baselines: Pointwise Scoring Pairwise Comparison IPO Skywork-Reward-V2-Qwen3-1.7B reward model Self-Reward Perplexity random selection condition number PCA 95% variance effective rank

Builds on: GRPO IPO Self-Rewarding Language Models DeepSeek-R1

Closest papers by meaning

• Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.90)
• RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.90)
• Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.89)
• Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.89)
• Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.88)
• Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.88)