Research map / Large language models / Instruction and fine-tuning
Optimize Incompatible Parameters Through Compatibility-aware Knowledge Integration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CKI, which fuses parameters of multiple trained models (static and dynamic recommenders, and language classifiers) into one same-sized model using local parameter-disagreement and global histogram-entropy compatibility scores, improving performance without added inference cost.
Problem
Deep networks contain incompatible parameters that hurt performance under varying data distributions; pruning targets efficiency rather than performance, and output ensembling multiplies inference cost.
Approach
Assess per-parameter compatibility across models using local parameter disagreement and global information entropy of parameter histograms, then splice parameters from several models via hard or soft weighting into a single model with the same parameter count.
Key results
- On Amazon-Beauty with static pre-trained models, CKI reaches NDCG@5 0.1409 vs 0.1250 for the best pruning baseline and 0.1137 for Model A.
- On SST-2, CKI achieves Acc 0.8544 and AUC 0.9161, versus 0.8532 Acc and 0.8530 AUC for output ensemble.
- Ablation on Douban-Book shows dual-perspective assessment gives NDCG@5 0.3823 versus 0.3779 without assessment, with soft splicing slightly outperforming hard splicing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| integrate parameters from two static models using CKI | recommendation models · Beauty, Book, Music, and Movielens | NDCG@5: Beauty: 0.1250 -> 0.1409; Book: 0.4089 -> 0.4095; Music: 0.3779 -> 0.3805; Movielens: 0.4464 -> 0.4480 | Helped |
| integrate static and dynamic model parameters using CKI | recommendation models · Beauty, Book, Music, and Movielens | NDCG@5: Beauty: 0.2046 -> 0.2296; Book: 0.4094 -> 0.4160; Music: 0.3772 -> 0.3929; Movielens: 0.4680 -> 0.4811 | Helped |
| integrate model parameters using CKI | language models · SST-2 | Acc: 0.8532 -> 0.8544 | Helped |
| integrate model parameters using CKI | language models · RTE | AUC: 0.6091 -> 0.5628 | Hurt |
| fine-tune the integrated model after CKI | CKI initialization · Douban Book | NDCG@5: 0.3823 -> 0.3831 | Helped |
| add local uncertainty assessment to global information content assessment (ablation) | CKI compatibility assessment · Douban Book | NDCG@5: 0.3803 -> 0.3823 | Helped |
| add global information content assessment to local uncertainty assessment (ablation) | CKI compatibility assessment · Douban Book | NDCG@5: 0.3797 -> 0.3823 | Helped |
| use global information content assessment instead of local uncertainty assessment (ablation) | CKI compatibility assessment · Douban Book | NDCG@5: 0.3797 -> 0.3803 | Helped |
| add local uncertainty assessment without global information content assessment (ablation) | CKI compatibility assessment · Douban Book | NDCG@5: 0.3779 -> 0.3797 | Helped |
| replace hard splicing with soft splicing (ablation) | CKI parameter splicing · Douban Book | HR@5: 0.5047 -> 0.5064 | Helped |
| replace hard splicing with soft splicing (ablation) | CKI parameter splicing · Douban Book | NDCG@5: 0.3806 -> 0.3805 | No clear effect |
| increase the number of integrated models up to 7 (ablation) | CKI · recommendation tasks | performance | Helped |
Surprising findings
- Pruning often performs worse than the original model, since it removes well-trained parameters along with incompatible ones.
- Parameter averaging alone frequently underperforms the individual base models, indicating naive merging is insufficient.
Limitations
- Performance gains over the best single model are sometimes modest (e.g., Beauty HR@5 gains small relative to ensembles on some metrics).
- Integrating too many models (beyond 7) degrades performance because the splicing network outgrows available data.
- Dual-perspective assessment and splicing network require training, and generalization analysis is only sketched under assumptions.
Experimental setup
Datasets: Amazon-Beauty Douban-Book Douban-Music Movielens-1M SST-2 RTE
Benchmarks: Amazon-Beauty Douban-Book Douban-Music Movielens-1M SST-2 RTE
Metrics: NDCG@5 HR@5 NDCG@10 HR@10 AUC UAUC Acc Recall F1
Baselines: Model Pruning Output Ensemble Parameter Averaging DUET DIN GRU4Rec SASRec RoBERTa
Builds on: lottery ticket hypothesis output ensemble model pruning
Closest papers by meaning
- Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging (NeurIPS 2025 · similarity 0.84)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.83)
- Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning (ICLR 2025 · similarity 0.83)
- Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective (ICLR 2024 · similarity 0.83)
- Semantic Convergence: Harmonizing Recommender Systems via Two-Stage Alignment and Behavioral Semantic Tokenization (AAAI 2025 · similarity 0.83)
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.