Research map / Architectures and efficiency / Mixture of experts
Bagging-Expert Network for Multi-Task Learning: A Depolarization Solution in Multi-Gate Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BEnet, a multi-task model that adds a feature-bagging layer and attention over context-feature subgroups to MMoE so experts stay diverse and avoid gate polarization.
Problem
Multi-gate mixture-of-experts (MMoE) suffers from polarization during training, where some expert gate weights collapse toward 0, wasting capacity and degrading multi-task performance in industrial recommendation.
Approach
Apply random-forest-style feature bagging (Bernoulli sampling of contextual feature subgroups, fixed once per network) so each expert and gate sees different inputs, then use target attention from user/item features over the bagged context subgroups to filter irrelevant information, encouraging experts to specialize and avoid polarization.
Key results
- On Company-A, BEnet's mean AUC is 0.8143 for CTR-type tasks versus best baseline values around 0.8108 (MMoE-dropout) in the reported table; on MultiMNIST BEnet mean accuracy is 0.9629 vs 0.9584 for MMoE-dropout.
- In an online A/B test on the Double 11 campaign (10% traffic), BEnet improved send rate by +8.5% and use rate by +9.3% over the ESMM online baseline.
- Gate weight plots show BEnet yields a smaller difference in convergence weights between experts than MMoE and MMoE-dropout on MultiMNIST and AliExpress-French.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BEnet instead of the strongest baseline | Multi-task learning · MultiMNIST | Mean ACC: 0.9584 -> 0.9629 | Helped |
| Use BEnet instead of the strongest baseline | Multi-task learning · AliExpress-French | Mean AUC: 0.8086 -> 0.8103 | Helped |
| Use BEnet instead of the strongest baseline | Multi-task learning · AliExpress-Spain | Mean AUC: 0.8126 -> 0.8143 | Helped |
| Use BEnet instead of the strongest baseline | Multi-task learning · Company-A | Mean AUC: 0.7821 -> 0.7848 | Helped |
| Use BEnet instead of the online baseline | Company-A coupon recommendation · Double 11 promotion campaign | send rate: +8.5% | Helped |
| Use BEnet instead of the online baseline | Company-A coupon recommendation · Double 11 promotion campaign | use rate: +9.3% | Helped |
| remove bagging from experts and gates (ablation) | BEnet · MultiMNIST | Mean ACC | Hurt |
Surprising findings
- Dropout-based depolarization was best at equalizing gate weights on AliExpress-Spain, so BEnet's depolarization advantage depends on the dataset.
- Removing bagging entirely (both bagging ratios 0) gave poor accuracy, consistent with polarization harming performance.
Limitations
- Gains over baselines on public datasets are modest (e.g., about 0.2 to 0.5 points in mean AUC/ACC).
- Bagging ratios require per-dataset fine-tuning; sensitivity analysis shows poor results without bagging.
- Public dataset experiments use general features rather than the user/item/context split on which the design is motivated.
Experimental setup
Datasets: MultiMNIST AliExpress (Spanish and French) Company-A coupon promotion logs
Benchmarks: MultiMNIST AliExpress Company-A
Metrics: accuracy AUC gate weight convergence send rate use rate
Baselines: PLE OMoE MMoE MMoE-dropout ESMM
Builds on: MMoE ESMM PLE dropout bagging / random forests
Closest papers by meaning
- Multi-Task Dense Prediction via Mixture of Low-Rank Experts (CVPR 2024 · similarity 0.85)
- Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Making (NeurIPS 2025 · similarity 0.84)
- Direct Routing Gradient (DRGrad): A Personalized Information Surgery for Multi-Task Learning (MTL) Recommendations (AAAI 2025 · similarity 0.84)
- STEM: Unleashing the Power of Embeddings for Multi-Task Recommendation (AAAI 2024 · similarity 0.84)
- MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (AAAI 2024 · similarity 0.83)
- RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.