Research map / Optimization / Nonconvex and smooth optimization
Propagation of Chaos for Mean-Field Langevin Dynamics and its Application to Model Ensemble
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Improves propagation-of-chaos bounds for mean-field Langevin dynamics via a uniform directional LSI, removing exponential LSI dependence from the particle error, and proposes PoC-based ensembling of MFNNs/LoRA adapters with guarantees.
Problem
Quantifying the finite-particle approximation error and optimization speed of mean-field Langevin dynamics (MFLD) for two-layer mean-field neural networks, where prior bounds scale exponentially with the regularization inverse due to LSI constants.
Approach
Introduce a uniform directional LSI on conditional Gibbs distributions and a defective LSI that bounds the particle approximation term by B/N independent of the LSI constant, yielding faster optimization rate exp(-2 alpha lambda t); then merge M independently trained N-neuron MFNNs (PoC-based ensemble) to reduce model approximation error.
Key results
- On commonsense reasoning with Llama2-7B, PoC merge reaches 82.74 average accuracy vs 78.69 for best single LoRA (32) and 61.28 for LoRA (256).
- With Llama3-8B, PoC merge averages 87.30 vs 85.26 for best LoRA (32) across eight datasets.
- Theory: the particle approximation term in MFLD bounds is B/N independent of LSI constant, and optimization rate exp(-2 alpha lambda t) improves on prior exponent tied to the LSI constant of the mean-field Gibbs distribution.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Merge 8 LoRA models with rank 32 instead of selecting the best single rank-32 model | Llama2-7B · Commonsense reasoning (average across 8 datasets) | accuracy: 78.69 -> 82.74 | Helped |
| Merge 8 LoRA models with rank 32 instead of selecting the best single rank-32 model | Llama3-8B · Commonsense reasoning (average across 8 datasets) | accuracy: 85.26 -> 87.30 | Helped |
| Use PoC merging of 8 rank-32 LoRA models instead of one rank-256 LoRA model at the same computational budget | Llama2-7B · Commonsense reasoning (average across 8 datasets) | accuracy: 61.28 -> 82.74 | Helped |
| Use PoC merging of 8 rank-32 LoRA models instead of one rank-256 LoRA model at the same computational budget | Llama3-8B · Commonsense reasoning (average across 8 datasets) | accuracy: 84.99 -> 87.30 | Helped |
Surprising findings
- Merging LoRA adapters trained in parallel (PoC merge) outperforms the best single LoRA run by about 4 points average on Llama2-7B and about 2 points on Llama3-8B.
Limitations
- Theory does not explain asymptotic behavior as lambda goes to 0; optimization slows exponentially.
- Particle term B/(lambda N) still depends on lambda via KL-based analysis.
- Empirical validation of the ensemble is on small synthetic settings and a single LoRA benchmark suite; code not yet released.
Experimental setup
Datasets: SIQA PIQA WinoGrande OBQA ARC-c ARC-e BoolQ HellaSwag synthetic double-circle data synthetic k multi-index data
Benchmarks: SIQA PIQA WinoGrande OBQA ARC-c ARC-e BoolQ HellaSwag
Metrics: accuracy sup-norm approximation error average accuracy
Baselines: LoRA (32, best) LoRA (256) Chen et al. 2024 PoC bound Kook et al. uniform-in-N LSI bound
Builds on: Chen et al. propagation of chaos for MFLD Holley-Stroock perturbation Mean-field Langevin dynamics LoRA
Closest papers by meaning
- Improved Particle Approximation Error for Mean Field Neural Networks (NeurIPS 2024 · similarity 0.94)
- Symmetric Mean-field Langevin Dynamics for Distributional Minimax Problems (ICLR 2024 · similarity 0.88)
- Mean-Field Langevin Dynamics for Signed Measures via a Bilevel Approach (NeurIPS 2024 · similarity 0.88)
- Mean-field Underdamped Langevin Dynamics and its Spacetime Discretization (ICML 2024 · similarity 0.87)
- Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamics (ICLR 2025 · similarity 0.86)
- Uniform-in-Time Weak Propagation of Chaos in Shallow Neural Networks (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.