Research map / Label-efficient and robust learning / Self-supervised and contrastive
Matrix Information Theory for Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Matrix-SSL, which reinterprets maximum entropy encoding as matrix uniformity loss and adds a matrix alignment loss on covariance matrices, improving non-contrastive SSL on ImageNet, COCO transfer, and LLM fine-tuning.
Problem
Non-contrastive self-supervised methods (SimSiam, Barlow Twins, MEC) are unified by a maximum entropy encoding loss but this loss does not explicitly separate the two branches, so an alignment term cannot be cleanly integrated.
Approach
Uses matrix information theory (matrix entropy, matrix KL, matrix cross-entropy) to show the MEC/TCR loss equals a matrix uniformity loss on the cross-covariance matrix, and adds a matrix alignment loss that aligns auto-covariance matrices of the two branches.
Key results
- Matrix-SSL reaches 71.1% top-1 linear evaluation on ImageNet at 100 epochs vs 70.6% for MEC and 66.5% for SimCLR/BYOL.
- On COCO detection, Matrix-SSL at 400 epochs gets 41.0 AP vs 39.8 for MEC and 37.9 for BYOL.
- Fine-tuning Llemma-MM 7B with matrix cross-entropy gives 72.3% on GSM8K, +3.1 over standard CE (69.2%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Matrix-SSL instead of MEC | ResNet50 self-supervised pretraining · ImageNet linear evaluation, 100/200/400 pretraining epochs | Top-1 accuracy: 70.6 -> 71.1; 71.9 -> 72.3; 73.5 -> 73.6 | Helped |
| Use Matrix-SSL instead of MEC | ResNet50 self-supervised pretraining · MS-COCO object detection | AP: 39.8 -> 41.0 | Helped |
| Use Matrix-SSL instead of MEC | ResNet50 self-supervised pretraining · MS-COCO instance segmentation | mask AP: 34.7 -> 35.6 | Helped |
| Fine-tune with Matrix-LLM loss instead of CE | Llemma-MM 7B · GSM8K | accuracy: 69.2 -> 72.3 (+3.1) | Helped |
| Fine-tune with Matrix-LLM loss instead of CE | Llemma-MM 7B · MATH | accuracy: 30.0 -> 30.2 (+0.2) | Helped |
| Set alignment loss ratio gamma to 0 instead of the default gamma=1 (ablation) | Matrix-SSL · ImageNet, 100-epoch linear evaluation | Top-1 accuracy: 71.1 -> 70.6 | Hurt |
| Set alignment loss ratio gamma to 0.3 instead of the default gamma=1 (ablation) | Matrix-SSL · ImageNet, 100-epoch linear evaluation | Top-1 accuracy: 71.1 -> 70.7 | Hurt |
| Set alignment loss ratio gamma to 0.5 instead of the default gamma=1 (ablation) | Matrix-SSL · ImageNet, 100-epoch linear evaluation | Top-1 accuracy: 71.1 -> 71.0 | Hurt |
| Set alignment loss ratio gamma to 0.6 instead of the default gamma=1 (ablation) | Matrix-SSL · ImageNet, 100-epoch linear evaluation | Top-1 accuracy: 71.1 -> 70.9 | Hurt |
| Set alignment loss ratio gamma to 1.3 instead of the default gamma=1 (ablation) | Matrix-SSL · ImageNet, 100-epoch linear evaluation | Top-1 accuracy: 71.1 -> 70.8 | Hurt |
| Set alignment loss ratio gamma to 1.5 instead of the default gamma=1 (ablation) | Matrix-SSL · ImageNet, 100-epoch linear evaluation | Top-1 accuracy: 71.1 -> 70.8 | Hurt |
| Use Taylor expansion order 3 instead of order 4 (ablation) | Matrix-SSL · ImageNet linear evaluation | Top-1 accuracy: 71.1 -> 70.9 | Hurt |
Surprising findings
- Setting the alignment loss weight gamma to 1 gave the best linear probing, while gamma=0 (uniformity only) gave 70.6%.
Limitations
- Auto-covariance uniformity loss left for future exploration
- Semi-supervised gains are mixed (significant at 1%, comparable at 10%)
- Multi-view augmentation effects are discussed but not evaluated
Experimental setup
Datasets: ImageNet MS-COCO MetaMathQA
Benchmarks: ImageNet linear evaluation MS-COCO object detection MS-COCO instance segmentation GSM8K MATH
Metrics: Top-1 accuracy AP AP50 AP75 accuracy
Baselines: SimCLR MoCo v2 BYOL SwAV SimSiam Barlow Twins VICReg MEC
Builds on: Maximum entropy encoding (MEC) Total coding rate (TCR) SimSiam BYOL Barlow Twins
Closest papers by meaning
- Information Flow in Self-Supervised Learning (ICML 2024 · similarity 0.87)
- PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignment (NeurIPS 2026 · similarity 0.85)
- Joint‑Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self‑Supervised Learning (NeurIPS 2025 · similarity 0.83)
- Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization (NeurIPS 2024 · similarity 0.83)
- Unveiling the Dynamics of Information Interplay in Supervised Learning (ICML 2024 · similarity 0.83)
- Cross-Entropy Is All You Need To Invert the Data Generating Process (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.