Research map / Label-efficient and robust learning / Recognition and long-tail
Hard Regularization to Prevent Deep Online Clustering Collapse without Data Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes combination assignment, a data-augmentation-free partition-support method for online deep clustering that regularizes hard batch assignments via a Bayesian prior and a greedy algorithm, avoiding collapse better than soft-assignment regularizers.
Problem
Online deep clustering tends to collapse into a degenerate solution where most or all points fall into one cluster; existing partition-support methods either need data augmentation or regularize soft assignments, which does not reliably prevent hard-assignment collapse.
Approach
Formulate batchwise hard cluster assignment as a Bayesian MAP problem with a multinoulli/combinatorial prior over cluster counts, derive a cost with a log(n_k+1) size penalty, and solve it greedily; the approximate objective is equivalent to maximizing mutual information between batch index and hard cluster labels.
Key results
- On CIFAR 10, combination assignment reaches 21.7% accuracy, 10.5 NMI and 5.4 ARI versus 16.7% accuracy for the next best method (SK), averaged over 5 runs.
- On FashionMNIST, the method reaches 59.3% accuracy and 55.3 NMI versus 25.5% and 20.9 NMI for the best baseline (ENT), with KL* near zero.
- Hard-assignment KL* is lowest for the proposed method on every dataset (e.g., 0.04 on CIFAR 10 vs 0.56 for SK), indicating less collapse of hard assignments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hard combination assignment rather than the strongest listed baseline | Online clustering · CIFAR 10 | accuracy: 18.6 (1.52) -> 21.7 (1.50) | Helped |
| Use hard combination assignment rather than the strongest listed baseline | Online clustering · CIFAR 100 | accuracy: 02.8 (0.34) -> 6.9 (0.17) | Helped |
| Use hard combination assignment rather than the strongest listed baseline | Online clustering · FashionMNIST | accuracy: 25.5 (6.16) -> 59.3 (4.16) | Helped |
| Use hard combination assignment rather than the strongest listed baseline | Online clustering · STL | accuracy: 18.5 (1.08) -> 24.2 (2.56) | Helped |
| Use hard combination assignment rather than the strongest listed baseline | Online clustering · RealDisp | accuracy: 23.5 (3.09) -> 57.0 (3.02) | Helped |
| Regularize hard assignments instead of soft assignments | Online clustering partition support · CIFAR 10 | hard KL*: 1.66 -> 0.04 | Helped |
| Regularize hard assignments instead of soft assignments | Online clustering partition support · CIFAR 100 | hard KL*: 4.47 -> 0.83 | Helped |
| Regularize hard assignments instead of soft assignments | Online clustering partition support · FashionMNIST | hard KL*: 2.30 -> 0.04 | Helped |
| Regularize hard assignments instead of soft assignments | Online clustering partition support · STL | hard KL*: 2.25 -> 0.09 | Helped |
| Regularize hard assignments instead of soft assignments | Online clustering partition support · RealDisp | hard KL*: 2.05 -> 0.05 | Helped |
| Use the modified variance-minimization method (VarM) instead of the original variance method (Var) (ablation) | Variance partition support · CIFAR 10 | accuracy: 11.8 (1.72) -> 20.8 (0.67) | Helped |
| Use the modified variance-minimization method (VarM) instead of the original variance method (Var) (ablation) | Variance partition support · FashionMNIST | accuracy: 10.0 (0.04) -> 37.4 (2.53) | Helped |
Surprising findings
- Soft-assignment entropy can be near maximal while hard assignments are fully collapsed, so regularizing mean soft assignments does not guarantee avoiding collapse.
- Entropy maximization over batch soft assignments occasionally still collapses to a single cluster.
Limitations
- Evaluated only on small-scale image datasets and one HAR dataset with a simple two-conv encoder and no hyperparameter tuning.
- Clustering performance remains low in absolute terms (e.g., ~21.7% on CIFAR 10), and the approximation to exact MI maximization relies on a dropped error term and a log(n+1) approximation.
- Greedy assignment is used instead of exact solution of the optimization problem.
Experimental setup
Datasets: CIFAR 10 CIFAR 100 FashionMNIST STL RealDisp
Benchmarks: CIFAR 10 CIFAR 100 FashionMNIST STL RealDisp
Metrics: accuracy NMI ARI KL* linear probe accuracy KNN accuracy
Baselines: Sinkhorn-Knopp (SK) marginal entropy maximization (ENT) sum of squares minimization (SS) concrete k-means (CKM) no regularization
Builds on: k-means Sinkhorn-Knopp concrete k-means
Closest papers by meaning
- Online Deep Clustering for Unsupervised Representation Learning (CVPR 2020 · similarity 0.88)
- You Can Trust Your Clustering Model: A Parameter-free Self-Boosting Plug-in for Deep Clustering (NeurIPS 2025 · similarity 0.85)
- From Logits to Hierarchies: Hierarchical Clustering made Simple (ICML 2025 · similarity 0.84)
- Breaking the Reclustering Barrier in Centroid-based Deep Clustering (ICLR 2025 · similarity 0.84)
- P$^2$OT: Progressive Partial Optimal Transport for Deep Imbalanced Clustering (ICLR 2024 · similarity 0.84)
- Interactive Deep Clustering via Value Mining (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.