Research map / Label-efficient and robust learning / Self-supervised and contrastive
Quality Diversity for Visual Pre-Training
Ruchika Chavhan, Henry Gouk, Da Li, Timothy Hospedales
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train an ensemble with diversity loss over augmentation invariances plus KL to a baseline | supervised ResNet50 on IN1K · 14 downstream tasks, linear readout | average rank: 2.7 -> 1.4 | Improved |
| QD pre-training versus supervised baseline on spatially sensitive regression | supervised ResNet50 · CelebA and Causal3D | R2 (%): 49.0 -> 61.5 (CelebA); 64.1 -> 72.8 (Causal3D) | Improved |
| QD pre-training versus baseline on CIFAR10 | supervised ResNet50 · CIFAR10 | accuracy (%): 90.4 -> 90.3 | No clear effect |
| use multihead ensemble with quality loss only instead of a diversity loss (ablation) | supervised ResNet50 · 14 downstream tasks | average rank: 2.7 -> 3.7 (Baseline vs Multihead) | Hurt |
| fuse ensemble with learned stacking weights instead of concatenation (ablation) | QD4V ensemble · Animal Pose (supervised) | R2 (%): 11.2 -> 12.5 | Improved |
| fuse ensemble with learned stacking weights instead of unweighted averaging (ablation) | QD4V ensemble · ALOI (MoCo IN1K) | R2 (%): 42.9 -> 47.4 | Improved |
| QD pre-training on few-shot regression | supervised ResNet50 · 300W s=0.05 | R2 (%): 20.5 -> 37.5 | Improved |
| QD pre-training on few-shot Flowers (5,1) versus diverse ensemble | supervised ResNet50 · Flowers (5,1) | accuracy (%): 77.3 -> 73.4 vs baseline | Hurt |
| QD pre-training with frozen backbone for dense tasks | supervised ResNet50 · Cityscapes and ADE20k | mIoU: 39.6 -> 41.0; 30.4 -> 33.8 | Improved |
| QD pre-training for MoCo detection on Pascal VOC | MoCo ResNet50 · Pascal VOC detection | AP: 54.2 -> 53.3 | Hurt |
| scale pre-training data (CLIP) instead of choosing invariances | CLIP RN50 versus QD4V · pose-sensitive tasks | downstream performance | Hurt |
Closest papers by meaning
- On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-training (NeurIPS 2023 · similarity 0.88)
- DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation (CVPR 2025 · similarity 0.87)
- Beyond Random Augmentations: Pretraining with Hard Views (ICLR 2025 · similarity 0.87)
- Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling (ICLR 2025 · similarity 0.86)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.86)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
