Research map / Label-efficient and robust learning / Recognition and long-tail
Neural Collapse in Multi-label Learning with Pick-all-label Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends neural collapse to multi-label classification under the pick-all-label cross-entropy loss, proving under the unconstrained feature model that global optima exhibit a multi-label ETF with tag-wise averaging, and uses this for an ONN prediction rule and parameter-efficient training.
Problem
Understanding the geometry of last-layer features and classifiers learned by deep networks for multi-label classification, where neural collapse has previously been studied only for multi-class classification.
Approach
Generalizes neural collapse to multi-label learning with the pick-all-label cross-entropy loss: single-label features form a simplex ETF, while higher-multiplicity features are scaled tag-wise sums of single-label feature means. Proves these are the only global optima of the unconstrained feature model and uses the structure to propose a one-nearest-neighbor decoder and fixed-ETF classifier traini
Key results
- ONN encoding beats one-vs-all on synthetic multi-label CIFAR-10 and SVHN in overall F1 and subset accuracy (e.g., SVHN F1 0.942 vs 0.937), while cutting prediction FLOPs from 1.068B to 0.0135B on SVHN.
- Fixing the classifier as a simplex ETF and reducing feature dimension to K saves up to about 20.7% of parameters (ResNet18 on MLab-MNIST) with comparable IoU.
- NC1, NC2, NC3 and the proposed NC_m metric all decrease toward zero during terminal-phase training across ResNet18/50 and VGG16/19 on the multi-label datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ONN instead of OvA encoding | ResNet18 with fixed ETF classifier · c10-L, c10-M, c10-S, and SVHN overall | F1-score: 0.896 -> 0.899; 0.885 -> 0.888; 0.857 -> 0.863; 0.937 -> 0.942 | Helped |
| Use ONN instead of OvA encoding | ResNet18 with fixed ETF classifier · c10-L, c10-M, c10-S, and SVHN overall | Subset zero-one accuracy: 0.787 -> 0.805; 0.778 -> 0.794; 0.740 -> 0.760; 0.831 -> 0.837 | Helped |
| Use ONN instead of OvA encoding | ResNet18 with fixed ETF classifier · c10-L, c10-M, c10-S, and SVHN | FLOPs(B): 2.46 -> 0.0307; 2.00 -> 0.0249; 1.54 -> 0.0192; 1.068 -> 0.0135 | Helped |
| Fix the classifier as ETF | ResNet18, ResNet50, VGG16, and VGG19 · MLab-MNIST | Test IoU (%): 99.5 -> 99.4; 99.4 -> 99.4; 99.5 -> 99.5; 99.5 -> 99.5 | No clear effect |
| Fix the classifier as ETF | ResNet18, ResNet50, VGG16, and VGG19 · MLab-Cifar10 | Test IoU (%): 87.7 -> 87.7; 88.9 -> 88.6; 86.9 -> 87.4; 88.7 -> 87.0 | Mixed |
Surprising findings
- Multi-label NC holds even when higher-multiplicity classes are imbalanced or missing, as long as single-label data are class-balanced.
- Empirically, the tag-wise average property still appears for samples whose multiplicity equals the total number of classes (M=K), beyond the theoretical assumption M<K.
Limitations
- The global optimality result relies on the unconstrained feature model and assumes d >= K-1 (benign landscape needs d > K).
- Theory assumes single-label (multiplicity-1) data are class-balanced; imbalance there is left to future work.
- Experiments use small synthetic multi-label datasets built from MNIST/CIFAR-10 and a limited MS-COCO subset with only 50 classes.
Experimental setup
Datasets: SVHN MNIST CIFAR-10 MS-COCO
Benchmarks: SVHN synthetic multi-label CIFAR-10 synthetic multi-label MNIST
Metrics: F1-score subset zero-one accuracy IoU NC1 NC2 NC3 NC_m
Baselines: one-vs-all (binary relevance) learned classifier
Builds on: neural collapse unconstrained feature model layer-peeled model
Closest papers by meaning
- MLC-NC: Long-Tailed Multi-Label Image Classification Through the Lens of Neural Collapse (AAAI 2025 · similarity 0.89)
- MaxSup: Overcoming Representation Collapse in Label Smoothing (NeurIPS 2025 · similarity 0.86)
- Double-Layer Hybrid-Label Identification Feature Selection for Multi-View Multi-Label Learning (AAAI 2024 · similarity 0.86)
- Addressing Multi-Label Learning with Partial Labels: From Sample Selection to Label Selection (AAAI 2025 · similarity 0.86)
- Generalized Neural Collapse for a Large Number of Classes (ICML 2024 · similarity 0.85)
- Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.