Research map / Large language models / Alignment and preferences

From “Weak” Signals to Strong Models: Preference Delta Aggregation with LoRA Merging

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Preference Delta Aggregation (PDA), which trains a LoRA adapter per weak-weaker preference pair and merges them with a new Grassmannian-aligned merging method (GAM), improving Qwen3-8B and Tülu3-8B on knowledge reasoning and agentic search.

Problem

Strong LLMs need high-quality supervision, which is scarce; naive training-based aggregation of multiple weak-weaker preference signals suffers from catastrophic forgetting and gradient conflicts, and existing LoRA merging suffers from directional interference.

Approach

Derive one preference delta per weak-weaker model pair as an independently trained LoRA adapter (DPO/ORPO), then merge via GAM: SVD-decompose each adapter, align left and right singular bases to a reference with orthogonal Procrustes rotations, and average the aligned components.

Key results

• PDA-GAM (3Δ) reaches 79.3 average EM on knowledge reasoning for Qwen3-8B vs 72.5 student baseline and 77.2 best single delta.
• PDA-GAM (4Δ) reaches 58.0 average F1 on agentic search for Qwen3-8B vs 50.7 student baseline and 53.7 best single delta.
• Sequential DPO (2Δ) scores 71.0 average EM on knowledge reasoning for Qwen3-8B, below the best single delta, indicating catastrophic forgetting.

Surprising findings

• A preference delta from ZeroSearch (7B over 3B) degraded the student by 1.3 F1 on MuSiQue, showing not all weak-weaker pairs help.
• Sequential and joint training of multiple preference deltas underperform the best single delta.

Limitations

• Gains rely on directionally diverse preference deltas; homogeneous pairs yield marginal gains.
• Only a limited set of weak-weaker model pairs was studied, so scaling behavior is unknown.
• Other aggregation strategies were not explored.

Experimental setup

Datasets: GSM8K MuSiQue

Benchmarks: MATH GPQA MMLU GSM8K NQ TriviaQA PopQA HotpotQA 2WikiMultiHopQA Bamboogle MuSiQue

Metrics: exact match token-level F1

Baselines: Student Baseline Llama-3.2 single delta Qwen3 single delta DeepSeek-R1 single delta Search-R1 single delta Sequential DPO Joint DPO Naive Averaging TIES-Merging KnOTS

Builds on: Delta Learning Hypothesis DPO ORPO LoRA TIES-Merging KnOTS

Closest papers by meaning

• Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model (ICLR 2025 · similarity 0.90)
• Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment (NeurIPS 2026 · similarity 0.89)
• Advantage-Guided Distillation for Preference Alignment in Small Language Models (ICLR 2025 · similarity 0.88)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.88)
• Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.88)
• TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.88)