Research map / Detection and segmentation / Open-set and few-shot detection
Near, far: Patch-ordering enhances vision foundation models' scene understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeCo, a self-supervised dense post-pretraining loss that enforces patch-level nearest-neighbor order consistency between student and teacher via differentiable sorting, improving DINOv2-register features for in-context and dense scene understanding.
Problem
Dense self-supervised features for in-context scene understanding (nearest-neighbor retrieval, linear/clustering segmentation, 3D correspondence) are limited; contrastive and object-level consistency losses give coarse signals, and nearest-neighbor retrieval is non-differentiable.
Approach
Apply differentiable sorting (bitonic sorting networks with soft swaps) to patch distances against reference patches, and enforce that the sorted nearest-neighbor permutation matrices of student and teacher agree via cross-entropy, as a post-pretraining step on top of already image-level pretrained ViTs.
Key results
- On in-context nearest-neighbor retrieval, NeCo (DINOv2R, ViT-B/14) reaches 69.0 mIoU on Pascal VOC at 1/128 data vs 59.1 for DINOv2R, and 44.5 vs 41.9 on ADE20K at full data.
- Frozen clustering on Pascal VOC (K=GT, ViT-S) improves CrIBo's 18.3 to 18.5 for NeCo(DINOv2R) and K=300 from 51.3 to 66.5.
- Linear segmentation with ViT-B/14 improves COCO-Stuff from 59.3 (DINOv2R) to 64.1 and Pascal VOC from 80.2 to 84.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| post-pretrain DINOv2R with NeCo | DINOv2R ViT-S/14 · Pascal VOC, K=500 clustering | mIoU: 64.5 -> 68.9 | Helped |
| post-pretrain DINOv2R with NeCo | DINOv2R ViT-S/14 · COCO-Things, linear segmentation | mIoU: 82.2 -> 82.3 | Helped |
| select both foreground and background patches instead of background patches (ablation) | NeCo · Pascal VOC linear segmentation | mIoU: 78.4 -> 78.9 | Helped |
| select both foreground and background patches instead of background patches (ablation) | NeCo · Pascal VOC in-context | mIoU: 60.5 -> 62.0 | Helped |
| add an EMA teacher (ablation) | NeCo · Pascal VOC linear segmentation | mIoU: 70.4 -> 78.9 | Helped |
| add an EMA teacher (ablation) | NeCo · Pascal VOC in-context | mIoU: 42.6 -> 62.0 | Helped |
| use all neighbors rather than the top 4 (ablation) | NeCo · Pascal VOC linear segmentation | mIoU: 74.4 -> 78.9 | Helped |
| use all neighbors rather than the top 4 (ablation) | NeCo · ADE20K in-context | mIoU: 19.7 -> 21.7 | Helped |
| train on COCO instead of ImageNet-100 (ablation) | NeCo · Pascal VOC in-context | mIoU: 55.8 -> 62.0 | Helped |
| increase batch size from 1 to 64 (ablation) | NeCo · Pascal VOC linear segmentation | mIoU: 76.0 -> 78.9 | Helped |
| select neighbors across batch images rather than from the same image (ablation) | NeCo · Multiple ablation metrics | performance: roughly 0.4% to 1% | Helped |
| remove the sorting component (ablation) | NeCo · Sorting ablation | performance | Hurt |
Surprising findings
- Applying NeCo to CrIBo's ViT-S gives only small linear gains on Pascal VOC (0.5%), while clustering gains are large.
- Starting from DINOv2 without registers, NeCo gave a mixed clustering result (e.g., K=GT Pascal VOC 15.5 vs DINOv2's 17.1 for ViT-S).
Limitations
- Gains on some settings are modest or mixed (e.g., DINOv2 without registers in some clustering metrics).
- Relies on pretrained image-level initialization; benefit depends on starting backbone.
- Batch size ablations suggest performance could improve further with larger batches, which were not tested.
Experimental setup
Datasets: COCO ImageNet-100 Pascal VOC 2012 ADE20K Pascal Context COCO-Stuff SPair-71k
Benchmarks: ADE20K Pascal VOC 2012 COCO-Things COCO-Stuff Pascal Context SPair-71k
Metrics: mIoU Recall@0.01
Baselines: CrIBo DINOv2R Hummingbird TimeT Leopart CrOC iBOT DINO COMUS SelfPatch
Builds on: DINOv2 DINO CrIBo Leopart Differentiable sorting networks
Closest papers by meaning
- 3DPoV: Improving 3D understanding via Patch Ordering on Videos (ICML 2026 · similarity 0.89)
- Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation (CVPR 2016 · similarity 0.88)
- LoCo: Learning 3D Location-Consistent Image Features with a Memory-Efficient Ranking Loss (NeurIPS 2024 · similarity 0.87)
- Building a Strong Pre-Training Baseline for Universal 3D Large-Scale Perception (CVPR 2024 · similarity 0.86)
- REOrdering Patches Improves Vision Models (NeurIPS 2025 · similarity 0.86)
- Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.