Research map / Label-efficient and robust learning / Self-supervised and contrastive
Learning to Rank Patches for Unbiased Image Redundancy Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LTRP, a self-supervised patch-ranking framework that scores visible patches by how much MAE reconstructions change when each patch is removed, then trains a ranker with ListMLE to select informative patches for unbiased image redundancy reduction.
Problem
Existing image redundancy reduction methods rely on category labels, causing a categorical inductive bias that discards content from unlabeled categories and hurts real-world generalization.
Approach
Use a frozen MAE at high (90%) masking ratio to derive pseudo semantic-density scores per visible patch from reconstruction changes when that patch is removed, then train a ranking model with a list-wise ListMLE loss on these self-supervised labels, so patch selection needs no class supervision.
Key results
- On MS-COCO multi-label classification, LTRP reaches 82.8% mAP at kr 0.75 and 81.8% at kr 0.5, beating the best unsupervised methods by 1.1% and 1.4% and the best supervised methods by 0.7%.
- On ImageNet-1K with ViT-T as ranker, LTRP speeds up inference by 18.9% to 32.7% on GPU with accuracy drops of only 0.1% to 0.5%.
- On unseen categories in PASCAL VOC and MS-COCO, LTRP surpasses all competitors on IoU, F1, recall and precision across keep ratios.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LTRP patch selection for multi-label classification | ViT classification on MS-COCO · MS-COCO, keep ratios 0.75 and 0.5 | mAP: 82.8%; 81.8%; outperforms previous best unsupervised methods by 1.1% and 1.4%, and supervised methods by 0.7% at both kr | Helped |
| Use LTRP to preserve patches for unseen categories | Image redundancy reduction methods · PASCAL VOC and MS-COCO object detection; MS-COCO segmentation | IoU, F1-score, recall, and precision | Helped |
| Use MAE-B instead of MAE-L or MAE-L-GAN to generate reconstruction pseudo-labels (ablation) | LTRP · ImageNet-100, keep ratio 0.5 | accuracy: 83.2; 83.0 -> 83.3 | Helped |
| Use a masking ratio of 0.8 instead of 0.9 (ablation) | LTRP · ImageNet-100, keep ratios 0.75 and 0.5 | accuracy: 84.4 -> 84.8; 83.3 -> 83.6 | Helped |
| Use PSNR instead of ℓ1 distance to score reconstruction changes (ablation) | LTRP · ImageNet-100, keep ratios 0.75 and 0.5 | accuracy: 84.4 -> 83.8; 83.3 -> 82.5 | Hurt |
| Use SSIM instead of ℓ1 distance to score reconstruction changes (ablation) | LTRP · ImageNet-100, keep ratios 0.75 and 0.5 | accuracy: 84.4 -> 83.6; 83.3 -> 79.9 | Hurt |
| Use ViT-T or ResNet50 instead of ViT-S as ranking model (ablation) | LTRP · ImageNet-100, keep ratios 0.75 and 0.5 | accuracy: 84.4 -> 84.5; 83.3 -> 83.4 | No clear effect |
| Add clustering to select homogeneous-region patches at clustering ratio 0.2 (ablation) | LTRP · ImageNet-100, keep ratios 0.75 and 0.5 | accuracy: 84.4 -> 84.6; 83.3 -> 83.7 | Helped |
| Set clustering ratio to 1 (ablation) | LTRP · ImageNet-100, keep ratios 0.75 and 0.5 | accuracy: 84.4 -> 83.6; 83.3 -> 82.7 | Hurt |
| Use RankNet instead of ListMLE ranking loss (ablation) | LTRP · ImageNet-100, keep ratios 0.75 and 0.5 | accuracy: 84.4 -> 82.9; 83.3 -> 79.5 | Hurt |
| Use pre-patch-normalized pixels instead of unnormalized pixels for reconstruction targets (ablation) | LTRP · ImageNet-100 | accuracy: 84.4 -> 81.5; 83.3 -> 75.7 | Hurt |
Surprising findings
- A point-wise ranking loss performs nearly as well as ListMLE (only 0.3% lower accuracy).
- Pixel normalization in the MAE reconstruction target hurts pseudo-label quality by 7.6% at kr 0.5 compared with unnormalized pixels.
Limitations
- Ablations run only on ImageNet-100 due to hardware constraints.
- Relies on the reconstruction quality of a pretrained MAE; large or GAN-based MAEs over-emphasize low-level texture and hurt accuracy.
- Extreme masking ratios create score plateaus that confuse the ranker.
Experimental setup
Datasets: ImageNet-1K ImageNet-100 MS-COCO PASCAL VOC ADE20K
Benchmarks: ImageNet-1K MS-COCO PASCAL VOC ADE20K ImageNet-100
Metrics: top-1 accuracy mAP IoU F1-score recall precision FLOPs inference speed
Baselines: GFNet Grad-CAM EViT IA-RED2 DynamicViT AdaViT DGE ToMe MoCo DINO
Builds on: masked autoencoder ListMLE EViT DPC-KNN
Closest papers by meaning
- Masked Scene Modeling: Narrowing the Gap Between Supervised and Self-Supervised Learning in 3D Scene Understanding (CVPR 2025 · similarity 0.84)
- Near, far: Patch-ordering enhances vision foundation models' scene understanding (ICLR 2025 · similarity 0.84)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.84)
- REOrdering Patches Improves Vision Models (NeurIPS 2025 · similarity 0.84)
- Collapse-Proof Non-Contrastive Self-Supervised Learning (ICML 2025 · similarity 0.83)
- No Pains, More Gains: Recycling Sub-Salient Patches for Efficient High-Resolution Image Recognition (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.