Research map / Label-efficient and robust learning / Semi-supervised learning
Unveiling the Dynamics of Information Interplay in Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes matrix mutual information ratio (MIR) and matrix entropy difference ratio (HDR) based on Neural Collapse to track representation–classifier information interplay, and adds them as loss terms to supervised and semi-supervised learning (FreeMatch variants).
Problem
Existing Neural Collapse analyses focus on feature/classifier similarity and largely ignore the information shared between sample representations and class classification heads; there is no normalized information metric to describe training dynamics and phenomena like linear mode connectivity, label
Approach
Define normalized matrix information metrics (MIR and HDR) on Gram matrices of sample features and classifier weights, derive their theoretical values under Neural Collapse (HDR→0, MIR→≈1), and use them as analysis tools and as auxiliary loss terms.
Key results
- Adding MIR/HDR losses to FreeMatch gives CIFAR-100 (400 labels) error 36.41% (MIR) vs 37.98% FreeMatch and CIFAR-10 (40 labels) 4.66% vs 4.90%.
- Fully supervised: CIFAR-10 95.35% baseline vs 95.57% with HDR loss; CIFAR-100 80.77% vs 80.96% with HDR loss.
- Theory: under Neural Collapse, HDR=0 and MIR = 1/(C-1) + (C-2)log(C-2)/((C-1)log(C-1)), approximately 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add mutual information loss | Fully supervised training · CIFAR-10 | accuracy: 95.35 -> 95.52 | Helped |
| Add entropy difference loss | Fully supervised training · CIFAR-10 | accuracy: 95.35 -> 95.57 | Helped |
| Add mutual information loss | Fully supervised training · CIFAR-100 | accuracy: 80.77 -> 80.81 | Helped |
| Add entropy difference loss | Fully supervised training · CIFAR-100 | accuracy: 80.77 -> 80.96 | Helped |
| Add mutual information loss to FreeMatch | FreeMatch · CIFAR-10 (10 labels) | error rate: 8.07 ± 4.24 -> 4.87 ± 0.66 | Helped |
| Add entropy difference loss to FreeMatch | FreeMatch · CIFAR-10 (10 labels) | error rate: 8.07 ± 4.24 -> 4.69 ± 0.16 | Helped |
| Add mutual information loss to FreeMatch | FreeMatch · CIFAR-100 (400 labels) | error rate: 37.98 ± 0.42 -> 36.41 ± 1.91 | Helped |
| Add entropy difference loss to FreeMatch | FreeMatch · STL-10 (40 labels) | error rate: 15.56 ± 0.55 -> 14.93 ± 3.28 | Helped |
| Vary label smoothing level | WideResNet models · CIFAR-10 and CIFAR-100 | accuracy | No clear effect |
| Prune model parameters and fine-tune the remaining subnetwork | Well-trained models · CIFAR-10 and CIFAR-100 | accuracy | No clear effect |
Surprising findings
- On CIFAR-10, linear mode connectivity fails at the default learning rate; interpolated models drop to random guessing, yet MIR rises in the 0.4–0.6 interpolation range; the authors say they find this anomaly hard to explain.
- In grokking, MIR first increases then decreases and HDR rises before dropping sharply after generalization, a two-phase pattern.
Limitations
- Dataset-level matrix entropy is approximated with batch matrix entropy due to compute constraints.
- Gains in fully supervised learning are slight, and some phenomena (CIFAR-10 linear mode connectivity anomaly) are not explained.
- Theoretical bounds rely on Neural Collapse holding, which is an idealized terminal-phase condition.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 STL-10 modular addition (p=113)
Benchmarks: CIFAR-10 CIFAR-100 STL-10
Metrics: accuracy error rate MIR HDR
Baselines: FreeMatch FixMatch FlexMatch SoftMatch OTMatch FreeMatch fully supervised MixMatch UDA
Builds on: Neural Collapse FreeMatch TorchSSL matrix-based Rényi entropy
Closest papers by meaning
- The Dynamics of Learning: A Random Matrix Approach (ICML 2018 · similarity 0.87)
- Cross-Entropy Is All You Need To Invert the Data Generating Process (ICLR 2025 · similarity 0.84)
- Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression (ICLR 2024 · similarity 0.84)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.84)
- Understanding and Leveraging the Learning Phases of Neural Networks (AAAI 2024 · similarity 0.84)
- Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.