Research map / Adversarial robustness and security / Attacks on text and graphs
Self-Consuming Generative Models with Adversarially Curated Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes self-consuming diffusion model retraining under adversarially curated (BT-model) user preference data, proves reward convergence depends on covariance with adversary's reward, and proposes gradient-based and heuristic label-flipping attacks evaluated on Gaussian, CIFAR-10 and CIFAR-100.
Problem
Self-consuming generative model retraining driven by user-curated synthetic data can be adversarially manipulated by competing platforms, misaligning the model from true user preferences.
Approach
Theoretically shows that the expected reward under adversarial curation evolves according to the covariance between the true reward and the adversary's reward, and formulates a bi-level covariance-minimizing label-flip attack on preference data learned from pairwise comparisons, solved via implicit gradients or heuristics.
Key results
- On CIFAR-10 with 20% flipped pairs, the gradient attack drives the one-round expected reward from 6.36 (benign) to 5.65 and 5.60; the Pareto heuristic reaches 5.47.
- Under gradient-based adversarial curation the model generates mostly the least-preferred classes (horse, ship, frog) on CIFAR-10, while benign curation shifts mass to airplane.
- Adding real data partially pulls the model toward p_data but does not steer it toward user preferences under adversarial curation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the gradient-based attack with different reward model architectures instead of benign curation | Self-consuming DDPM · CIFAR-10 | Avg. R: 6.3606 -> 5.6460 | Helped |
| Use the gradient-based attack with identical reward model architectures instead of benign curation | Self-consuming DDPM · CIFAR-10 | Avg. R: 6.3606 -> 5.5959 | Helped |
| Use the multi-objective Pareto heuristic attack instead of benign curation | Self-consuming DDPM · CIFAR-10 | Avg. R: 6.3606 -> 5.4740 | Helped |
| Use the reward-based heuristic selecting pairs by |R(x)-R(z)| instead of benign curation | Self-consuming DDPM · CIFAR-10 | Avg. R: 6.3606 -> 5.5982 | Helped |
| Use the reward-based heuristic selecting pairs by max(|R(x)|,|R(z)|) instead of benign curation | Self-consuming DDPM · CIFAR-10 | Avg. R: 6.3606 -> 5.5612 | Helped |
| Use the multi-objective Pareto heuristic instead of either reward-based heuristic | Self-consuming DDPM · CIFAR-10 | Avg. R: 5.5982 -> 5.4740; 5.5612 -> 5.4740 | Helped |
Surprising findings
- Adding real data to the retraining mix does not defend against targeted adversarial curation, only partially mitigating it.
- Random label flipping produces an initial reward increase before fluctuating, showing it is less damaging than targeted attacks.
Limitations
- Theory assumes each model update converges to the global optimum of training.
- Attack assumes a known success rate kappa of curation, which the attacker may not know in practice.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 synthetic 8-mode Gaussian mixture
Benchmarks: CIFAR-10 CIFAR-100 synthetic Gaussian
Metrics: class proportion of generated samples expected reward E[r(x)] of generated samples
Baselines: Benign curation random attack R-Based heuristic Pareto multi-objective heuristic
Builds on: self-consuming generative models (Ferbach et al.) generalized Bradley-Terry model DDPM NSGA-II
Closest papers by meaning
- Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences (NeurIPS 2024 · similarity 0.92)
- Self-Consuming Generative Models with Co-Evolving Human Preferences (NeurIPS 2026 · similarity 0.90)
- On the Stability of Iterative Retraining of Generative Models on their own Data (ICLR 2024 · similarity 0.88)
- A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops (ICLR 2025 · similarity 0.88)
- Stability and Diversity of Networked Self-Consuming Generative Ecosystems (NeurIPS 2026 · similarity 0.88)
- Towards Theoretical Understandings of Self-Consuming Generative Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.