Research map / Label-efficient and robust learning / Self-supervised and contrastive
Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| multi-mask training (reuse teacher targets across M masked versions) (ablation) | data2vec 2.0 vision · ImageNet-1K reduced setup, bsz=64 | top-1 accuracy: +4.6% for M=16 vs M=2 | Helped |
| multi-mask training effect diminishes with larger batch size (ablation) | data2vec 2.0 vision · ImageNet-1K reduced setup | top-1 accuracy | Mixed |
| adding global CLS loss (ablation) | data2vec 2.0 vision · ImageNet-1K reduced setup | top-1 accuracy: small improvement | Helped |
| adding MAE pixel regression loss to contextualized target loss (ablation) | data2vec 2.0 vision · ImageNet-1K reduced setup | top-1 accuracy | No clear effect |
| pixel regression loss only instead of contextualized targets (ablation) | data2vec 2.0 vision · ImageNet-1K reduced setup | top-1 accuracy: substantial drop | Hurt |
| inverse block masking instead of block masking or random masking (B=3) (ablation) | data2vec 2.0 vision · ImageNet-1K reduced setup | top-1 accuracy | Helped |
| alibi relative position bias in speech feature encoder (ablation) | data2vec 2.0 speech · Librispeech / Libri-light fine-tuning | word error rate | Helped |
| data2vec 2.0 vs wav2vec 2.0 for speech recognition efficiency | data2vec 2.0 speech · Libri-light speech recognition | pre-training wall clock time: 10.6x less time at comparable WER | Helped |
| data2vec 2.0 vs RoBERTa for NLP efficiency | data2vec 2.0 NLP · GLUE | pre-training speed and average GLUE dev accuracy: comparable accuracy at 1.8x speed | No clear effect |
| data2vec 2.0 vs MAE for ImageNet pretraining efficiency | data2vec 2.0 vision · ImageNet-1K | pre-training time at matched accuracy: 16.4x speed-up, 83.7 vs 83.6 | Helped |
| increasing pre-training dataset size (ablation) | data2vec 2.0 vision base models · ImageNet-1K subsampled pretraining | top-1 accuracy | Helped |
Closest papers by meaning
- data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (ICML 2022 · similarity 0.94)
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (NeurIPS 2020 · similarity 0.90)
- No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations (NeurIPS 2024 · similarity 0.86)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.86)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.86)
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.