Research map / Label-efficient and robust learning / Domain adaptation and generalization
Connect Later: Improving Fine-tuning for Robustness with Targeted Augmentations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Connect Later, which pretrains with generic augmentations (SwAV or masked autoencoding) and then fine-tunes with domain-knowledge targeted augmentations, improving OOD accuracy on four real-world domain adaptation tasks.
Problem
Pretraining with generic augmentations does not consistently improve out-of-distribution performance under unsupervised domain adaptation; standard fine-tuning after pretraining can fail to beat training from scratch on labeled source data.
Approach
After pretraining with generic augmentations, fine-tune the pretrained encoder on labeled source data using targeted augmentations that match the target distribution in a feature space where domains differ (e.g., copy-paste backgrounds, stain jitter, redshifting), so pretrained within-domain representations are combined with cross-domain connectivity from augmentation.
Key results
- Connect Later reaches 79.90% OOD accuracy on AstroClassification versus 77.40% for Avocado and 67.84% for standard fine-tuning.
- On iWildCam-WILDS, Connect Later achieves 36.9 OOD Macro F1 versus 31.2 for standard fine-tuning and 34.5 for ICON.
- On Camelyon17-WILDS with DenseNet121, Connect Later achieves 94.9% OOD average accuracy versus 93.8% for ICON.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune pretrained representations with redshift-targeted augmentations rather than standard fine-tuning | Informer MAE · AstroClassification OOD Test | accuracy: 67.84 ± 0.70 -> 79.90 ± 0.60 | Helped |
| Fine-tune pretrained representations with redshift-targeted augmentations rather than standard fine-tuning | Informer MAE · Redshifts OOD Test | RMSE: 0.277 ± 0.004 -> 0.247 ± 0.005 | Helped |
| Use Connect Later instead of ERM + targeted augs | ResNet-50 · iWildCam-WILDS OOD Test | Macro F1: 36.1 ± 0.7 -> 36.9 ± 0.7 | Helped |
| Use Connect Later instead of ICON | DenseNet121 · Camelyon17-WILDS OOD Test | average accuracy: 93.8 ± 0.3 -> 94.9 ± 0.4 | Helped |
| Scale the model from 21M to 69M parameters (ablation) | Connect Later · AstroClassification ID Test | accuracy: 78.47 -> 80.38 | Helped |
| Scale the model from 21M to 69M parameters (ablation) | Connect Later · AstroClassification OOD Test | accuracy: 79.49 -> 80.55 | Helped |
| Add linear probing before fine-tuning (ablation) | Connect Later · AstroClassification OOD Test | accuracy: 78.6 -> 79.49 | Helped |
| Add linear probing before fine-tuning (ablation) | ERM + targeted augs · AstroClassification OOD Test | accuracy: 68.43 -> 67.07 | Hurt |
| Vary MAE pretraining masking percentage across 20%, 40%, 60%, 80%, and 90% (ablation) | Connect Later · AstroClassification | ID and OOD accuracy | No clear effect |
| Use 90% instead of weaker MAE masking percentages (ablation) | Connect Later · AstroClassification OOD Test | accuracy | Hurt |
Surprising findings
- Contrastive pretraining followed by standard fine-tuning barely improves OOD Macro F1 on iWildCam-WILDS (30.4 to 31.2) despite prior evidence that contrastive pretraining aids domain adaptation.
- LP-FT decreased OOD accuracy by 1.4% for ERM+targeted augs (no pretraining) while improving Connect Later.
Limitations
- Targeted augmentations require manual domain knowledge for each distribution shift.
- Results on pretraining benefit are inconsistent across datasets, and the connectivity explanation is only partly tested.
- Compute and hardware details are not reported.
Experimental setup
Datasets: iWildCam-WILDS Camelyon17-WILDS AstroClassification (adapted from PLAsTiCC 2019) Redshifts (adapted from PLAsTiCC 2019) ImageNet
Benchmarks: iWildCam-WILDS Camelyon17-WILDS AstroClassification Redshifts
Metrics: Macro F1 average accuracy accuracy RMSE
Baselines: ERM standard fine-tuning ERM + targeted augs Self-Training Avocado DANN CORAL Noisy Student ICON
Builds on: SwAV LP-FT masked autoencoding augmentation graph theory of contrastive learning (HaoChen et al.) Avocado
Closest papers by meaning
- ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.85)
- CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.85)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.85)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.85)
- Accelerating Augmentation Invariance Pretraining (NeurIPS 2024 · similarity 0.84)
- DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.