Research map / Label-efficient and robust learning / Recognition and long-tail
Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that low-rank sigmoid output layers in multi-label classification leave exponentially many label combinations unargmaxable, and proposes a truncated DFT output layer (with slack columns) that guarantees all label sets with up to k active labels are argmaxable while using fewer trainable parameters.
Problem
Bottlenecked sigmoid output layers (features fewer than labels) in multi-label classification cannot predict exponentially many label combinations regardless of input, and some of these may be meaningful outputs that occur in data.
Approach
Formalise argmaxability for MLC, verify it per label set with a Chebyshev linear program, and construct a truncated Discrete Fourier Transform output layer whose weight matrix lies in the totally positive Grassmannian (all maximal minors nonzero with same sign), which guarantees every k-active label assignment is argmaxable; add learnable slack columns and a bias initialisation trick for faster tr
Key results
- BSL with d=200 on MIMIC-III yields unargmaxable test label sets (3370 of 3371 1-argmaxable), while the DFT layer makes all test label sets argmaxable at d=200 and above.
- On BioASQ, a DFT layer with d=100 surpasses the F1@10 of a BSL with d=200, using about 50% fewer trainable parameters.
- DFT converges about 25% faster in training time than the BSL on BioASQ and OpenImages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BSL with DFT at d=25 | CNN multi-label classifier · MIMIC-III test | ε-argmaxable assignments: 229 -> 3371 | Helped |
| Replace BSL with DFT at d=50 | CNN multi-label classifier · MIMIC-III test | ε-argmaxable assignments: 1515 -> 3371 | Helped |
| Replace BSL with DFT at d=100 | CNN multi-label classifier · MIMIC-III test | ε-argmaxable assignments: 3137 -> 3371 | Helped |
| Replace BSL with DFT at d=25 | PubMedBERT multi-label classifier · BioASQ test | ε-argmaxable assignments: 880 -> 9995 | Helped |
| Replace BSL with DFT at d=50 | PubMedBERT multi-label classifier · BioASQ test | ε-argmaxable assignments: 6498 -> 10000 | Helped |
| Replace BSL with DFT at d=25 | TResnet multi-label classifier · OpenImages test | ε-argmaxable assignments: 2758 -> 10000 | Helped |
| Replace BSL with DFT at d=50 | TResnet multi-label classifier · OpenImages test | ε-argmaxable assignments: 8439 -> 10000 | Helped |
| Add 16 randomly initialized slack columns to the DFT layer (ablation) | DFT output layer · Synthetic label assignments as label count increases | ε-argmaxability | Helped |
| Initialize the DFT projection bias to target probability k/n (ablation) | DFT output layer · BioASQ and OpenImages training | training time to convergence: about 25% faster | Helped |
Surprising findings
- Exponentially many label combinations (fraction of 2^n) are unargmaxable for a BSL with d much smaller than n, and some real test label sets are unargmaxable.
- Some DFT label combinations are argmaxable but not epsilon-argmaxable at small d, because regions shrink with more labels.
Limitations
- Guarantees hold only for label sets with at most k active labels; k must be bounded per dataset.
- Small bottleneck widths make DFT regions tiny, so some label sets are only argmaxable without epsilon margin and LP numerical precision limits verification.
- Analysis restricted to sigmoid bottleneck layers; other output layer families not studied.
Experimental setup
Datasets: MIMIC-III BioASQ task A 2021 OpenImages v6
Benchmarks: MIMIC-III BioASQ task A OpenImages v6
Metrics: Prec@8 F1@k eps-argmaxable count 1-argmaxable count validation loss
Baselines: bottlenecked sigmoid layer (BSL)
Builds on: sign rank cyclic polytope trigonometric moment curve
Closest papers by meaning
- PD-Sparse : A Primal and Dual Sparse Approach to Extreme Multiclass and Multilabel Classification (ICML 2016 · similarity 0.89)
- MLC-NC: Long-Tailed Multi-Label Image Classification Through the Lens of Neural Collapse (AAAI 2025 · similarity 0.87)
- Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels (ICLR 2025 · similarity 0.87)
- Self-Paced Unified Representation Learning for Hierarchical Multi-Label Classification (AAAI 2024 · similarity 0.87)
- Toward Robustness in Multi-Label Classification: A Data Augmentation Strategy against Imbalance and Noise (AAAI 2024 · similarity 0.86)
- Forming Auxiliary High-confident Instance-level Loss to Promote Learning from Label Proportions (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.