Research map / Label-efficient and robust learning / Semi-supervised learning

ABC-Align: Prediction-Powered Alignment with Adaptive Bias Control

NeurIPS 2026 · Label-efficient and robust learning › Semi-supervised learning · approach: Confidence-based pseudo-label selection · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ABC-Align, a biased-but-lower-variance prediction-powered gradient estimator with an online controller over two mixing parameters, applied to reward model training, DPO and GRPO with scarce human labels.

Problem

Post-training LLMs when human preference labels are scarce and pseudo labels from a teacher (RLAIF-style) are abundant but biased; unbiased prediction-powered gradient estimators are stuck with a Theta(1/n) variance floor.

Approach

Relax unbiasedness with a two-parameter ABC gradient estimator (a, b) whose bias is (1-a)B; an online corrected-norm controller (projected AdaGrad on a surrogate with a split-batch cross-term correction) or a finite-sample plug-in (for GRPO) tunes (a, b) to trade bias for variance.

Key results

• On Skywork RM at label fraction 0.01 and flip rate 0.20, ABC-Align reaches 0.77 vs 0.70 for PP-SSL and 0.64 for labeled-only.
• DPO on UltraFeedback (Llama-3.2-3B) reaches 0.72 at label fraction 0.05 and 0.75 at 0.30, versus PP-SSL 0.69-0.72.
• On GRPO GSM8K (Llama-3.2-3B), ABC-Align improves from 0.721 at label fraction 0.05 to 0.752 at 0.30, with PP-SSL 1.5-2 points behind.

Surprising findings

• Naive combined pseudo-label pooling falls below labeled-only when teacher agreement is at chance (random-init head), as shown in the real biased teacher table.

Limitations

• The online controller's EMA smoothing of the bias estimate breaks the theoretical unbiasedness assumption, so theory does not strictly cover the experiments.
• The plug-in pair for GRPO is not predictable, so the exact MSE identity does not hold for the deployed update.
• Experiments use a frozen backbone for reward modeling and synthetic flip-rate teacher noise for most sweeps, with small-scale models (3B).

Experimental setup

Datasets: Skywork-Reward-80K UltraFeedback Binarized GSM8K-style math prompts RewardBench PPE

Benchmarks: RewardBench PPE GSM8K

Metrics: held-out pairwise accuracy pass@1 preference accuracy

Baselines: Labeled-only Naive combined PP-SSL Pseudo-label-only Doubly Robust

Builds on: prediction-powered inference PP-SSL DPO GRPO doubly robust estimation

Closest papers by meaning

• Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.90)
• RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.89)
• Exploring Intrinsic Alignments Within Text Corpus (AAAI 2025 · similarity 0.89)
• Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.89)
• Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment (ICLR 2024 · similarity 0.89)
• Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment (NeurIPS 2025 · similarity 0.89)