Research map / Learning theory and algorithms / Prediction and decision losses
Annealed Multiple Choice Learning: Overcoming limitations of Winner-takes-all with annealing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Annealed Multiple Choice Learning (aMCL), which replaces the hard min in WTA with a temperature-controlled softmin (deterministic annealing), and analyzes its training dynamics via rate-distortion and phase transitions.
Problem
Winner-takes-all training in multiple choice learning is greedy and can converge to arbitrarily suboptimal local minima, with hypothesis collapse where some hypotheses are unused.
Approach
Replace the hard min of WTA with a softmin over hypotheses (Boltzmann assignment with a decreasing temperature schedule, stop-gradient on assignments), so that all hypotheses receive gradient early on and the hypotheses split progressively as temperature cools; connects this to entropy-constrained alternating minimization, rate-distortion theory and phase transitions.
Key results
- On UCI, aMCL achieves lower distortion than MCL on most datasets, e.g., Protein 0.77 vs 0.80 and Kin8nm 6.81e-4 vs 1.00e-3.
- On WSJ0-mix 2-speaker separation, aMCL reaches 16.85 SI-SDR vs 16.30 for MCL and 16.70 for Relaxed-WTA, with lower seed variance than MCL.
- aMCL matches PIT on 2- and 3-speaker separation (16.85 vs 16.88; 10.00 vs 10.01) while reducing matching complexity from O(m^3) to O(mn).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use annealed soft assignments instead of hard Winner-takes-all assignments | MCL · UCI regression datasets | Distortion | Mixed |
| Use annealed soft assignments instead of hard Winner-takes-all assignments | MCL · UCI regression datasets | RMSE | Mixed |
| Use annealed soft assignments instead of hard Winner-takes-all assignments | MCL · WSJ0-mix eval, 2 speakers | PIT SI-SDR: 16.30 $\pm$ 0.59 -> 16.85 $\hfill\pm$ 0.13 | No clear effect |
| Use annealed soft assignments instead of hard Winner-takes-all assignments | MCL · WSJ0-mix eval, 3 speakers | PIT SI-SDR: 10.06 $\hfill\pm$ 0.21 -> 10.00 $\hfill\pm$ 0.21 | No clear effect |
| Use annealed soft assignments instead of hard Winner-takes-all assignments | Relaxed-WTA · WSJ0-mix eval, 2 speakers | PIT SI-SDR: 16.70 $\hfill\pm$ 0.08 -> 16.85 $\hfill\pm$ 0.13 | Helped |
| Use annealed soft assignments instead of hard Winner-takes-all assignments | Relaxed-WTA · WSJ0-mix eval, 3 speakers | PIT SI-SDR: 9.43 $\pm$ 0.21 -> 10.00 $\hfill\pm$ 0.21 | Helped |
| Use annealed assignments instead of hard WTA assignments | MCL · WSJ0-mix speech separation across training seeds | inter-seed standard deviation of PIT SI-SDR | Helped |
Surprising findings
- Relaxed-WTA is biased toward the global barycenter, which hurts quantization distortion even though it can help RMSE.
- MCL shows hypothesis collapse for some training seeds on speech separation, while aMCL is more robust to initialization.
Experimental setup
Datasets: UCI regression benchmark WSJ0-mix synthetic Gaussian mixtures
Benchmarks: UCI regression benchmark WSJ0-mix
Metrics: distortion RMSE SI-SDR
Baselines: MCL Relaxed-WTA PBP MC Dropout Deep Ensembles PIT
Builds on: Multiple Choice Learning Relaxed-WTA deterministic annealing PIT
Closest papers by meaning
- Winner-takes-all for Multivariate Probabilistic Time Series Forecasting (ICML 2025 · similarity 0.86)
- XTC: Head-Aware Sampling by Excluding Top Choices (NeurIPS 2026 · similarity 0.84)
- Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.84)
- Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions (ICLR 2025 · similarity 0.83)
- Uncertainty Herding: One Active Learning Method for All Label Budgets (ICLR 2025 · similarity 0.83)
- MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.