Research map / Detection and segmentation / CNN features
Three Guidelines You Should Know for Universally Slimmable Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train one slimmable model with US3L rather than individually trained BYOL models | ResNet-18 and ResNet-50 · ImageNet, 1.0x and 0.25x | linear accuracy (%): 54.0 -> 56.9; 34.9 -> 40.7; 68.1 -> 68.4; 50.9 -> 57.7 | Helped |
| Use US3L rather than individually trained MoCov3 | ViT-Tiny and ViT-Small · CIFAR-10, 1.0x and 0.25x | linear accuracy (%): 82.6 -> 86.0; 68.0 -> 80.2; 88.0 -> 90.3; 75.5 -> 85.5 | Helped |
| Add momentum targets to MSE distillation (ablation) | US3L with MSE base loss and MSE distillation · CIFAR-100, ResNet-18, 1.0x | linear accuracy (%): 57.5 -> 64.7 | Helped |
| Add an auxiliary distillation head (ablation) | US3L with MSE base and distillation losses and momentum targets · CIFAR-100, ResNet-18, 1.0x | linear accuracy (%): 64.7 -> 65.4 | Helped |
| Use InfoNCE rather than MSE as the base loss, with MSE distillation and momentum targets (ablation) | US3L · CIFAR-100, ResNet-18, 1.0x | linear accuracy (%): 65.4 -> 66.9 | Helped |
| Add an auxiliary distillation head (ablation) | US3L with InfoNCE base and MSE distillation and momentum targets · CIFAR-100, ResNet-18, 1.0x | linear accuracy (%): 66.9 -> 67.7 | Helped |
| Remove distillation and momentum targets from MSE-based training (ablation) | US3L with MSE base loss · CIFAR-100, ResNet-18 | linear accuracy (%): model collapses (-) | Hurt |
| Add dynamic sampling (ablation) | US3L · CIFAR-100, ResNet-18, 1.0x | linear accuracy (%): 67.7 -> 68.6 | Helped |
| Add group regularization (ablation) | US3L · CIFAR-100, ResNet-18, 1.0x | linear accuracy (%): 67.7 -> 68.6 | Helped |
| Add max-first sampling (ablation) | US3L dynamic sampling · CIFAR-100, ResNet-18 | expected forward number: 3T -> 2.5T | Helped |
| Gradually reduce the smallest sampled width (ablation) | US3L dynamic sampling · CIFAR-100, ResNet-18 | linear accuracy (%): 67.7 -> 68.5 | Helped |
| Add group regularization to individually trained BYOL or SimCLR (ablation) | BYOL and SimCLR · CIFAR-100, ResNet-18 | linear accuracy (%): 66.8 -> 66.0; 66.5 -> 65.7 | Hurt |
Closest papers by meaning
- Slimmable Neural Networks (ICLR 2019 · similarity 0.86)
- Effective Self-supervised Pre-training on Low-compute Networks without Distillation (ICLR 2023 · similarity 0.85)
- Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data (ICLR 2021 · similarity 0.84)
- Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.83)
- Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization (NeurIPS 2024 · similarity 0.82)
- Probing the Mid-level Vision Capabilities of Self-Supervised Learning (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.