Research map / Vision-language and multimodal / Human and embodied modeling
Towards flexible perception with visual memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a retrieval-based visual memory that classifies images by kNN search over pre-trained embeddings (DinoV2, CLIP) with a new RankVoting aggregation, showing flexible data addition, unlearning, pruning, and interpretability.
Problem
Deep vision models store knowledge in weights, making it hard to add, remove, or interpret knowledge after training; the paper asks whether an explicit visual memory can offer flexible perception.
Approach
Decompose classification into fixed pre-trained image similarity plus fast nearest-neighbor search over a database of (embedding, label) pairs; aggregate neighbors with a parameter-free rank-based power-function vote (RankVoting) that avoids the decaying accuracy of softmax/plurality voting as k grows. Capabilities such as unlearning (delete memory entries), memory pruning, and class addition foll
Key results
- RankVoting with DINOv2 ViT-L/14 plus Gemini 1.5 Flash re-ranking of 50 neighbors reaches 88.5% ImageNet val top-1, vs 83.5% kNN and 86.3% linear probe.
- With DINOv2 ViT-L/14, adding 64 NINCO OOD classes to memory keeps IN-val at 83.6% (RankVoting) while reaching 87.4% on NINCO without training.
- Human accuracy on predicting ImageNet-A model decisions rises from 56% (black-box) to 83% with five nearest-neighbor images from the memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Re-rank the 50 nearest neighbors with Gemini 1.5 Flash | DinoV2 ViT-L14 visual memory · ImageNet-1K val | accuracy: 83.5% -> 88.5% | Helped |
| Add RankVoting instead of SoftmaxVoting | DinoV2 ViT-L14 visual memory · ImageNet-1K val | accuracy: 83.6 -> 83.6 | No clear effect |
| Add RankVoting instead of SoftmaxVoting | CLIP ViT-L14 visual memory · ImageNet-1K val | accuracy: 79.6 -> 79.9 | Helped |
| Use JFT pseudo-label memory rather than ImageNet-train memory | DinoV2 ViT-L14 visual memory · ImageNet-A | accuracy: 58.8 -> 61.1 | Helped |
| Use JFT pseudo-label memory rather than ImageNet-train memory | DinoV2 ViT-L14 visual memory · ImageNet-R | accuracy: 62.8 -> 73.7 | Helped |
| Use JFT pseudo-label memory rather than ImageNet-train memory | DinoV2 ViT-L14 visual memory · ImageNet-Sketch | accuracy: 61.5 -> 68.0 | Helped |
| Use JFT pseudo-label memory rather than ImageNet-train memory | DinoV2 ViT-L14 visual memory · ImageNet-V2 | accuracy: 75.6 -> 77.6 | Helped |
| Use JFT pseudo-label memory rather than ImageNet-train memory | DinoV2 ViT-L14 visual memory · ImageNet-ReaL | accuracy: 87.1 -> 88.2 | Helped |
| Add soft memory pruning (ablation) | DinoV2 ViT-L14 + RankVoting · ImageNet-1K val | accuracy: 83.6 -> 84.1 (+0.5%) | Helped |
| Add hard memory pruning (ablation) | DinoV2 ViT-L14 + RankVoting · ImageNet-1K val | accuracy: 83.6 -> 83.7 | Helped |
| Add soft memory pruning (ablation) | DinoV2 ViT-L14 + SoftmaxVoting · ImageNet-1K val | accuracy: 83.6 -> 83.9 (+0.3%) | Helped |
| Add NINCO samples and labels to the memory | DinoV2 ViT-L14 RankVoting · ImageNet validation | accuracy: 83.6 -> 83.6 | No clear effect |
Surprising findings
- Existing aggregation methods (plurality, distance, softmax voting) show decaying accuracy as more neighbors are added, while RankVoting improves then plateaus.
- Performance stays nearly unchanged even with about 60% randomly corrupted labels in memory.
Limitations
- Only image classification evaluated; extension to detection, segmentation, and generation left open.
- Relies on a fixed pre-trained embedding; strong distribution shifts may require updating it.
- Trade-off between flexibility and accuracy: memory approach sometimes falls below linear probe on static classification.
Experimental setup
Datasets: ImageNet-1K JFT-3B subset NINCO iNaturalist 2021
Benchmarks: ImageNet-1K ImageNet-A ImageNet-R ImageNet-Sketch ImageNet-V2 ImageNet-ReaL NINCO iNaturalist 2021
Metrics: top-1 accuracy error rate forgetting quality accuracy gap human prediction accuracy
Baselines: plurality voting distance voting softmax voting linear probe kNN classifier NeurIPS 2023 machine unlearning challenge winner
Builds on: DINOv2 CLIP ScaNN
Closest papers by meaning
- VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models (CVPR 2026 · similarity 0.88)
- Separating Knowledge and Perception with Procedural Data (ICML 2025 · similarity 0.88)
- Learning from Memory: Non-Parametric Memory Augmented Self-Supervised Learning of Visual Features (ICML 2024 · similarity 0.87)
- NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning (NeurIPS 2026 · similarity 0.87)
- VL2Lite: Task-Specific Knowledge Distillation from Large Vision-Language Models to Lightweight Networks (CVPR 2025 · similarity 0.86)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.