AI Paper Atlas

Research map / Large language models / Alignment and preferences

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

Davit Melikidze, Marian Schneider, Jessica Lam, Martin Wertich, Ido Hakimi, Barna Pásztor, Andreas Krause

ICML 2026 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
select pairs with DRTS instead of random sampling (ablation)ActiveUltraFeedback with DPO · 4 downstream benchmarks mean (UltraFeedback prompts)mean score delta vs base: +0.046 -> +0.127Improved
select pairs with DeltaUCB instead of random sampling (ablation)ActiveUltraFeedback with DPO · RewardBench 2score delta vs base: +0.278 -> +0.339Improved
use dueling bandit selection (DTS, MaxMinLCB, InfoMax) instead of random (ablation)ActiveUltraFeedback · 4 downstream benchmarks mean (DPO)mean score delta vs base: +0.046 (Random) -> +0.023 (DTS), +0.016 (MaxMinLCB)Hurt
use DeltaQwen fixed small-vs-large pairing instead of active selection for reward modeling (ablation)preference data generation · RewardBench 2score delta vs base: +0.100 (DeltaQwen) vs +0.278 (Random)Hurt
use DRTS or DeltaUCB selection with a small subset of data (ablation)DPO fine-tuning · downstream mean, UltraFeedback promptsmean score deltaImproved
switch DPO to IPO or SimPO with DeltaQwen data (ablation)DeltaQwen · mean downstream scoremean score deltaHurt
use DRTS or DeltaUCB on other prompt sources (Skywork, Combined, Tulu 3) (ablation)ActiveUltraFeedback · DPO mean scoremean score deltaImproved
use DRTS or DeltaUCB instead of original Skywork/Combined dataset for reward modeling (ablation)ActiveUltraFeedback · RewardBench 2, Skywork and Combined promptsscore deltaHurt
use DRTS or DeltaUCB with IPO and SimPO instead of DPO (ablation)ActiveUltraFeedback · mean downstream score, IPO and SimPOmean score deltaImproved
use probabilistic judge scoring (expectation over score tokens) instead of discrete generation (ablation)LLM judge annotation · UltraFeedback judged pairswin rate: 75.36% -> 76.70%Improved
enable explicit reasoning in the LLM judge (ablation)LLM judge annotation · UltraFeedback judged pairswin rate: 76.70% -> 73.54%Hurt
use probabilistic instead of discrete scoring (ablation)LLM judge annotation · RewardBench 2mean: 0.698 -> 0.744Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy