Research map / Large language models / Alignment and preferences
Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks training-free model merging against data-mixture methods for balanced helpfulness, honesty, and harmlessness (3H) DPO alignment of Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2, and proposes RESM, an outlier-weighted, sparsity-adaptive rank task-singular-vector merging method.
Problem
Balancing helpfulness, honesty, and harmlessness when aligning LLMs; data mixture methods rely on expert knowledge and suffer conflicting signals, while model merging for 3H alignment is underexplored.
Approach
Extends task singular vector merging (TSVM) with 3-sigma outlier-aware reweighting of singular values to suppress preference-noise outliers, and layer-wise sparsity-adaptive rank selection instead of a fixed top-k rank, enabling end-to-end merging of individually DPO-trained 3H models.
Key results
- On Llama3-8B-Instruct, RESM reaches +14.97% normalized gain vs. +10.16% for the best data mixture method (Hummer) and +13.00% for TSVM.
- On Mistral-7B-Instruct-v0.2, RESM reaches +3.36% normalized gain vs. +1.53% for Hummer and +2.45% for TSVM.
- In the continual setting on Llama3, RESM achieves +14.56% normalized gain, above the best sequential DPO stage (+13.00%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RESM instead of TSVM | 3H model merging for Llama-3-8B-Instruct · 3H static optimization | Norm_Gain: 13.00% -> 14.97% | Helped |
| Use RESM instead of Hummer data mixture | 3H alignment for Llama-3-8B-Instruct · 3H static optimization | Norm_Gain: 10.16% -> +14.97% | Helped |
| Use RESM instead of TSVM | 3H model merging for Mistral-7B-Instruct-v0.2 · 3H static optimization | Norm_Gain: +2.45% -> +3.36% | Helped |
| Use RESM instead of Hummer data mixture | 3H alignment for Mistral-7B-Instruct-v0.2 · 3H static optimization | Norm_Gain: +1.53% -> +3.36% | Helped |
| Use RESM instead of TSVM | 3H model merging for Llama-3-8B-Instruct · 3H continual optimization | Norm_Gain: +13.47% -> +14.56% | Helped |
| Add outlier weighting (ablation) | RESM on Llama-3-8B-Instruct · 3H static optimization | Norm_Gain: +13.60% -> +14.97% | Helped |
| Add sparsity-adaptive rank selection (ablation) | RESM on Llama-3-8B-Instruct · 3H static optimization | Norm_Gain: +13.71% -> +14.97% | Helped |
| Add outlier weighting (ablation) | RESM on Mistral-7B-Instruct-v0.2 · 3H static optimization | Norm_Gain: +2.89% -> +3.36% | Helped |
| Add sparsity-adaptive rank selection (ablation) | RESM on Mistral-7B-Instruct-v0.2 · 3H static optimization | Norm_Gain: +3.12% -> +3.36% | Helped |
Surprising findings
- Helpfulness-only DPO training improved honesty and harmlessness, while honesty and harmlessness training weakened each other on Llama3.
- Several sparsification-based merges (e.g., DELLA-Ties vs. DARE-Ties) showed opposite trends across Llama3 and Mistral, indicating unstable results.
Limitations
- Merging evaluated only with training-free methods; training-based and MoE-based merging excluded from the comparison.
- Conflict-probability bound is a theoretical argument with simplifying assumptions (random unit vectors).
- Mixed-task results show small gains on Mistral (+3.36%) and some metrics degrade (e.g., Honesty normalized relative to base).
Experimental setup
Datasets: 10 preference datasets (5 annotation dimensions) SimPO preference data
Benchmarks: GSM8K Math ARC-E ARC-C MMLU MBPP-Plus HumanEval-Plus MT-Bench HaluEval-Wild SaladBench OR-Bench
Metrics: accuracy MT-Bench score no-hallucination rate safe rate refusal rate normalized gain
Baselines: Weight Average Rewarded Soup Model Stock Task Arithmetic TIES-Merging DARE DELLA Breadcrumbs TSVM Hummer ArmoRM Heuristic data mixture
Builds on: TSVM DPO SimPO TIES-Merging DARE
Closest papers by meaning
- $\texttt{Model-GLUE}$: Democratized LLM Scaling for A Large Model Zoo in the Wild (NeurIPS 2024 · similarity 0.90)
- Activation-Informed Merging of Large Language Models (NeurIPS 2025 · similarity 0.90)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.90)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.90)
- AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization (CVPR 2025 · similarity 0.89)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.