Research map / Video / Video segmentation and tracking
Your Image is My Video: Reshaping the Receptive Field via Image-To-Video Differentiable AutoAugmentation and Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DAS, a differentiable auto-augmentation search that turns augmented copies of an image into a pseudo-video processed by a 2D backbone with temporal shift (GSF), reshaping the spatial receptive field for classification and segmentation.
Problem
Automatic data augmentation methods are slow (RL or grid search) or depend on hand-designed search spaces, and it is unclear how augmentation-induced temporal receptive field expansion affects spatial receptive field in image tasks.
Approach
Relax the choice of affine/photometric transformations with softmax weights (DARTS-style) to search a cell that produces a sequence of transformed images; treat that sequence as a video, process it with a temporal shift network (GSF), and select final operations by perturbation-based importance rather than largest weight.
Key results
- ResNet-50 with DAS + GSF reaches 79.45% top-1 on ImageNet, versus 76.30% baseline, and ResNet-152 reaches 80.13%.
- DeepLab-v3 with DAS reaches 87.00% mIOU on Pascal-VOC-2012 versus 85.70% for the baseline.
- On Cityscapes, DAS with ResNeSt-200 reaches 85.10% mIOU, reported as best result.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DAS-selected transformations and temporal fusion | ResNet-50 · ImageNet | Top-1 accuracy: 76.30 -> 79.45 | Helped |
| Add DAS-selected transformations and temporal fusion | DeepLabv3 · PASCAL-VOC-2012 | mIOU: 85.70 -> 87.00 | Helped |
| Add augmentation without temporal fusion (ablation) | PSPNet · PASCAL-VOC-2012 | mIOU: 85.40 -> 85.51 | No clear effect |
| Add identity-copy temporal replicas (ablation) | PSPNet · PASCAL-VOC-2012 | mIOU: 85.40 -> 83.40 | Hurt |
| Add searched transformations and temporal fusion (ablation) | PSPNet · PASCAL-VOC-2012 | mIOU: 85.40 -> 86.10 | Helped |
| Replace GSF with TSM (ablation) | PSPNet · PASCAL-VOC-2012 | mIOU: 86.10 -> 85.86 | Hurt |
| Replace GSF with a 3D CNN (ablation) | PSPNet · PASCAL-VOC-2012 | mIOU: 86.10 -> 77.00 | Hurt |
| Replace DAS with AutoAugment (ablation) | PSPNet · PASCAL-VOC-2012 | mIOU: 87.00 -> 82.15 | Hurt |
| Replace DAS with RandAugment (ablation) | PSPNet · PASCAL-VOC-2012 | mIOU: 87.00 -> 84.00 | Hurt |
| Use a transformation genotype searched on another dataset (ablation) | DAS · PASCAL-VOC-2012 | mIOU: 86.10 -> 77.25 | Hurt |
| Use a transformation genotype searched on another dataset (ablation) | DAS · Cifar-10 | accuracy: 95.12 -> 93.14 | Hurt |
Surprising findings
- Stacking identical image copies as a video (Replica) lowers accuracy (e.g., 90.11% vs 94.12% on CIFAR-10), indicating that augmentation diversity and fusion are both required.
- Randomly chosen transformation genotypes cause large performance drops compared with DAS-searched ones (e.g., Pascal-VOC 77.25 vs 86.10).
Limitations
- Memory footprint grows with the number of generated frames, limiting search to 8-frame videos.
- Over-segmentation observed on Pascal-VOC-2012 in some cases.
- Only 2D backbones with temporal shift were tested; transformer backbones left for future work.
Experimental setup
Datasets: ImageNet CIFAR-10 CIFAR-100 Tiny-ImageNet Pascal-VOC-2012 Cityscapes
Benchmarks: ImageNet CIFAR-10 CIFAR-100 Tiny-ImageNet Pascal-VOC-2012 Cityscapes
Metrics: top-1 accuracy mIOU parameters FLOPs
Baselines: AutoAugment RandAugment FFC dilated convolution DeepLabv3 PSPNet ViT-L-32 3D CNN
Builds on: DARTS GSF TSM AutoAugment RandAugment
Closest papers by meaning
- Effective Data Augmentation With Diffusion Models (ICLR 2024 · similarity 0.85)
- Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.84)
- Multi-Scale Representations by Varying Window Attention for Semantic Segmentation (ICLR 2024 · similarity 0.84)
- QuARF: Quality-Adaptive Receptive Fields for Degraded Image Perception (AAAI 2025 · similarity 0.84)
- Receptive Fields As Experts in Convolutional Neural Architectures (ICML 2024 · similarity 0.84)
- VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.