Research map / Label-efficient and robust learning / Self-supervised and contrastive
When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loop
Yang Zhang, Xiukun Wei, Xueru Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase human-curated fraction for model theta while its data is heavily cross-model | two-model self-consuming loop (theta) · CIFAR-10 diffusion, A4 | Jp(theta*) reward | Hurt |
| increase human curation for theta when cross-model data fraction is small (<=0.3) | two-model self-consuming loop (theta) · CIFAR-10 diffusion, A1-A3 and A5 | Jp(theta*) reward | Improved |
| increase curation data for model phi when its training data comes entirely from conflicting model theta | two-model self-consuming loop (phi) · CIFAR-10 diffusion | Jq(phi*) reward | Hurt |
| reduce influence of theta on phi (lower cross-model fraction) while curating theta | two-model self-consuming loop (phi) · CIFAR-10 diffusion, A4 | Jq(phi*) reward | Improved |
| strong coupling between models (A6) | two-model self-consuming loop · CIFAR-10 diffusion, A6 | rewards | Hurt |
| increase coupling scale t with fixed curation lambda_H=0.4 | Gaussian coupled system · Gaussian models, t in [0.05,1] | Jq(phi*) derivative | Hurt |
| train only on cross-model data (preference domain mismatch) | Qwen2.5-0.5B summarization/paraphrase models · Qwen2.5-0.5B, lambda cross=1 | target-domain reward | No clear effect |
| increase sample size n used for model updates | Gaussian experiment estimators · Gaussian models, n in [4,64] | statistical error vs theory | Improved |
Closest papers by meaning
- Self-Consuming Generative Models with Co-Evolving Human Preferences (NeurIPS 2026 · similarity 0.91)
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.90)
- Stability and Diversity of Networked Self-Consuming Generative Ecosystems (NeurIPS 2026 · similarity 0.88)
- Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences (NeurIPS 2024 · similarity 0.88)
- Self-Consuming Generative Models with Adversarially Curated Data (ICML 2025 · similarity 0.87)
- More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
