Research map / Label-efficient and robust learning / Recognition and long-tail
TdAttenMix: Top-Down Attention Guided Mixup
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TdAttenMix, a CutMix variant that uses a top-down attention module (label-conditioned ViT classifier weights added to bottom-up self-attention) to pick crop regions and set label mixing ratios, and evaluates it on classification, transfer, robustness and an ARISTO gaze-based inconsistency metric.
Problem
CutMix and saliency-based variants select crops using bottom-up saliency, which often highlights regions inconsistent with the image label, producing image-label inconsistency in mixed training samples.
Approach
Augment bottom-up self-attention of a ViT with a top-down signal derived from the label's final-layer classifier weights (balanced by a factor sigma), use the resulting task-adaptive attention to choose max-min attention crop regions, and mix labels by combining area ratio and attention ratio.
Key results
- TdAttenMix reaches 82.36% top-1 on CIFAR-100 with ResNet-18 and 81.19% on ImageNet-1k with DeiT-S, surpassing the listed prior methods such as AutoMix (82.04 and 80.78).
- On the ARISTO gaze-based inconsistency metric, TdAttenMix scores 18.4 versus 26.2 for CutMix and 18.9 for SaliencyMix.
- With DeiT-S, TdAttenMix improves ADE20k mIoU from 31.6 to 33.2 and Jaccard index on PASCAL VOC from 29.2 to 32.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TdAttenMix instead of AutoMix | ResNet-18 · CUB-200 | top-1 accuracy (%): 79.87 -> 80.71 | Helped |
| Use TdAttenMix instead of the strongest reported baseline | DeiT-S · ImageNet-1k | top-1 accuracy (%): 81.08 -> 81.19 | Helped |
| Train the backbone with TdAttenMix | DeiT-S · ADE20k semantic segmentation | mIoU (%): +1.6 | Helped |
| Train the backbone with TdAttenMix | DeiT-S · ADE20k semantic segmentation | mAcc (%): +2.5 | Helped |
| Train the backbone with TdAttenMix | DeiT-S · PASCAL VOC WSAS | Segmentation JI (%): +3.3 | Helped |
| Use task-adaptive top-down attention rather than bottom-up attention only (σ=0) (ablation) | TdAttenMix · Image-label inconsistency (ARISTO) | inconsistency: 19.0 -> 18.4 | Helped |
| Set balanced attention factor σ to 0 rather than the full-method setting σ=1 (ablation) | TdAttenMix · CUB-200 | top-1 accuracy (%): 80.71 -> 80.31 | Hurt |
| Set balanced attention factor σ to 0.5 rather than σ=1 (ablation) | TdAttenMix · CUB-200 | top-1 accuracy (%): 80.71 -> 80.60 | Hurt |
| Set balanced attention factor σ to 2 rather than σ=1 (ablation) | TdAttenMix · CUB-200 | top-1 accuracy (%): 80.71 -> 80.29 | Hurt |
| Set balanced attention factor σ to 3 rather than σ=1 (ablation) | TdAttenMix · CUB-200 | top-1 accuracy (%): 80.71 -> 79.89 | Hurt |
| Set balanced attention factor σ to 4 rather than σ=1 (ablation) | TdAttenMix · CUB-200 | top-1 accuracy (%): 80.71 -> 79.50 | Hurt |
| Set area-attention label-mixing weight β to 0 rather than 0.5 (ablation) | TdAttenMix · CUB-200 | top-1 accuracy (%): 80.71 -> 80.20 | Hurt |
Surprising findings
- The top-down balance factor sigma=1 is best, while larger values (2-4) degrade accuracy, and the bottom-up-only setting performs close to SaliencyMix on the inconsistency metric.
Limitations
- Image-label inconsistency is measured only on the ARISTO gaze dataset, limiting the scope of the quantitative analysis.
- Several strong baselines (TransMix, TokenMix, SMMix) are ViT-specific, so comparisons across architectures are incomplete.
- Hyperparameter details and extra comparisons are deferred to the supplementary material.
Experimental setup
Datasets: CIFAR-100 Tiny-ImageNet CUB-200 ImageNet-1k ADE20k PASCAL VOC 2012 ImageNet-A ImageNet-O ARISTO
Benchmarks: CIFAR-100 Tiny-ImageNet CUB-200 ImageNet-1k ADE20k PASCAL VOC 2012 ImageNet-A ImageNet-O ARISTO
Metrics: top-1 accuracy top-2 accuracy mIoU mAcc Jaccard index calibration error AURRA AUPR inconsistency
Baselines: Vanilla SaliencyMix MixUp CutMix ManifoldMix SmoothMix AttentiveMix PuzzleMix Co-Mixup GridMix TransMix TokenMix
Builds on: CutMix SaliencyMix SMMix Max-Min Attention Region Mixing OpenMixup
Closest papers by meaning
- MixPro: Data Augmentation with MaskMix and Progressive Attention Labeling for Vision Transformer (ICLR 2023 · similarity 0.87)
- Provable Benefit of Cutout and CutMix for Feature Learning (NeurIPS 2024 · similarity 0.83)
- Adversarial AutoMixup (ICLR 2024 · similarity 0.83)
- DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models (CVPR 2024 · similarity 0.83)
- Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model (CVPR 2024 · similarity 0.82)
- Learning from Observer Gaze: Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition (CVPR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.