Research map / Architectures and efficiency / CNN design
No Pains, More Gains: Recycling Sub-Salient Patches for Efficient High-Resolution Image Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dual-Buffer Patch Selection (DBPS) that recycles sub-salient patch embeddings in a no-gradient buffer, paired with a dual-attention MIL aggregator and random patch drop, to train high-resolution image recognition models with more patches at low cost.
Problem
High-resolution image recognition (HRIR) methods that select a few salient patches miss broadly distributed context, while encoding more patches in gradient mode exceeds GPU memory and training time budgets.
Approach
A dual buffer keeps the top-M salient patch embeddings (re-embedded with gradients) and recycles the next S sub-salient embeddings without gradients; a dual-attention MIL uses the salient aggregate as a query to suppress background in sub-salient patches, and random patch drop further cuts training cost.
Key results
- On CQU-BPDD, DBPSformer reaches 83.2% accuracy with 1.2 GB VRAM and 111 ms, versus IPS-Transformer's 82.2% at 8.2 GB and 279 ms when trained on 108 patches.
- On CAMELYON16, DBPSformer achieves 99.3% AUC with 1.5 GB VRAM, above IPS-Transformer's 98.4% AUC at 14.3 GB.
- Ablation on CQU-BPDD shows the dual-buffer strategy alone raises accuracy from 80.1% to 82.2%, with dual-attention MIL and random patch drop adding further gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dual-buffer patch selection to IPSformer (ablation) | IPSformer · CQU-BPDD | ACC: 80.1 -> 82.2 | Helped |
| add dual-buffer patch selection to IPSformer (ablation) | IPSformer · Functional Map of the World | ACC: 77.1 -> 77.9 | Helped |
| add dual-attention MIL to dual-buffer patch selection (ablation) | DBPSformer · CQU-BPDD | ACC: 82.2 -> 82.8 | Helped |
| add dual-attention MIL to dual-buffer patch selection (ablation) | DBPSformer · Functional Map of the World | ACC: 77.9 -> 78.4 | Helped |
| add random patch drop with ratio 0.1 to IPS baseline (ablation) | IPSformer · CQU-BPDD | ACC: 80.1 -> 80.8 | Helped |
| add random patch drop with ratio 0.1 to IPS baseline (ablation) | IPSformer · Functional Map of the World | ACC: 77.1 -> 77.3 | Helped |
| add random patch drop with ratio 0.2 to IPS baseline (ablation) | IPSformer · CQU-BPDD | ACC: 80.1 -> 81.1 | Helped |
| add random patch drop with ratio 0.2 to IPS baseline (ablation) | IPSformer · Functional Map of the World | ACC: 77.1 -> 77.4 | Helped |
| add random patch drop with ratio 0.3 to IPS baseline (ablation) | IPSformer · CQU-BPDD | ACC: 80.1 -> 81.2 | Helped |
| add random patch drop with ratio 0.3 to IPS baseline (ablation) | IPSformer · Functional Map of the World | ACC: 77.1 -> 77.2 | No clear effect |
| add random patch drop with ratio 0.1 to DBPSformer (ablation) | DBPSformer · CQU-BPDD | ACC: 82.8 -> 82.9 | Helped |
| add random patch drop with ratio 0.1 to DBPSformer (ablation) | DBPSformer · Functional Map of the World | ACC: 78.4 -> 78.6 | Helped |
Surprising findings
- Training with few patches but evaluating with many patches gives the worst accuracy for IPS-Transformer, so simply adding inference patches does not help.
Limitations
- Sub-salient patches are used without gradients, so the encoder only indirectly benefits from them.
- Random patch drop ratio and buffer sizes require tuning; gains vary across datasets (e.g., some settings degrade on Swedish Traffic Signs).
Experimental setup
Datasets: CQU-BPDD Functional Map of the World (fMoW) Swedish Traffic Signs Recognition CAMELYON16 DDR MAME
Benchmarks: CQU-BPDD Functional Map of the World (fMoW) Swedish Traffic Signs Recognition CAMELYON16 DDR MAME
Metrics: accuracy AUC maximum GPU memory (VRAM, GB) batch training runtime (ms)
Baselines: IPS-Transformer DPSformer Zoom-In Zoom-In+ TopMIL DSMIL-LC CLAM DeepMIL KIPRN WSPLIN
Builds on: IPS-Transformer DS-MIL Differential Patch Selection ResNet
Closest papers by meaning
- Iterative Patch Selection for High-Resolution Image Recognition (ICLR 2023 · similarity 0.86)
- ESNet: Evolution and Succession Network for High-Resolution Salient Object Detection (ICML 2024 · similarity 0.85)
- HR-NAS: Searching Efficient High-Resolution Neural Architectures With Lightweight Transformers (CVPR 2021 · similarity 0.85)
- Beyond Image Super-Resolution for Image Recognition with Task-Driven Perceptual Loss (CVPR 2024 · similarity 0.85)
- MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolution (NeurIPS 2024 · similarity 0.85)
- SparseFormer: Sparse Visual Recognition via Limited Latent Tokens (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.