Research map / Detection and segmentation / Open-set and few-shot detection
Separating Knowledge and Perception with Procedural Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a DINO ViT-S embedding model only on procedural shader-based data (Shaders KML Mixup) and applies it with a visual memory of real reference embeddings for similarity, classification, and segmentation, achieving full separation from real-world training data.
Problem
Visual memory (KNN over embeddings) separates knowledge from the classifier, but the feature embeddings are still trained on real images, so knowledge and perception remain entangled and unlearning/privacy guarantees are incomplete.
Approach
Train the embedding model with DINO only on procedurally generated shader images (with a new KMeans-mask-based mixing process, Shaders KML, and Shaders KML Mixup), then insert real-world knowledge only as a database of reference embeddings, enabling exact unlearning and per-sample privacy checks.
Key results
- Shaders KML achieves 82.4% NIGHTS alignment, within 0.9% of the Places-trained model (83.3%).
- Shaders KML Mixup KNN accuracy on ImageNet-1K is 37.88% vs 47.30% for Places, and beats Places on Flowers102 (75.20 vs 59.51) and CUB200 (27.08 vs 19.09).
- On COCO, procedural PCA R^2 reaches 55.9 (S. KML) vs 62.1 for Places and 63.7 for ImageNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on Shaders KML Mixup instead of Places | ViT-S with visual-memory KNN · Flowers102 | KNN classification accuracy: 59.51 -> 75.20 | Helped |
| Train on Shaders KML Mixup instead of Places | ViT-S with visual-memory KNN · CUB200 | KNN classification accuracy: 19.09 -> 27.08 | Helped |
| Train on Shaders KML Mixup instead of Places | ViT-S with visual-memory KNN · Food101 | KNN classification accuracy: 47.78 -> 48.70 | Helped |
| Train on Shaders KML Mixup instead of Places | ViT-S with visual-memory KNN · ImageNet-1K | KNN classification accuracy: 47.30 -> 37.88 | Hurt |
| Train on Shaders KML instead of Places | ViT-S PCA features · COCO | R^2: 62.1 -> 55.9 | Hurt |
| Use Shaders KML instead of Shaders Mixup (ablation) | Procedural embedding training · ImageNet-1K | linear top-1 accuracy: 44.8 -> 47.1 | Helped |
| Add Mixup to Shaders KML (ablation) | Procedural embedding training · CUB200 | KNN classification accuracy: 24.02 -> 27.08 | Helped |
| Add Mixup to Shaders KML (ablation) | Procedural embedding training · COCO | R^2: 55.9 -> 53.7 | Hurt |
| Apply data-driven KMeans mixing masks instead of constant Mixup masks (ablation) | Shaders Mixup procedural embeddings · ImageNet-1K | linear top-1 accuracy: 44.8 -> 47.1 | Helped |
| Train on procedural Shaders KML instead of Places | ViT-S visual-similarity embeddings · NIGHTS-Val | human alignment: 0.8331 $\pm$ 0.0090 -> 0.8244 $\pm$ 0.0092 | No clear effect |
Surprising findings
- Procedural models beat the Places real-data model on fine-grained classification (Flowers, CUB, Food).
- Procedural models struggle with KNN semantic segmentation due to overly local, non-gestalt representations; most models, real or procedural, fail gestalt tests except Closure.
Limitations
- Procedural models lag real-data models on ImageNet-1K classification and COCO segmentation.
- Procedural embeddings have overly local representations and fail to join parts of the same object, causing incorrect nearest-neighbour searches.
- Memory storage scales with the number of examples, and KNN inference is costlier than the linear classifier on the original code.
Experimental setup
Datasets: Shaders StyleGAN procedural data ImageNet Places NIGHTS CUB200 Flowers102 Food101 ImageNet-1K COCO Ade20k CelebA
Benchmarks: NIGHTS-Val CUB200 Flowers102 Food101 ImageNet-1K COCO Ade20k MedMNIST CelebA gestalt segmentation
Metrics: 2AFC alignment accuracy KNN classification accuracy R^2 of PCA features vs labels privacy: fraction of non-private training samples
Baselines: ImageNet-trained ViT-S Places-trained ViT-S Shaders Shaders Mixup StyleGAN procedural Random init SIFT RGB PSNR SSIM
Builds on: DINO Mixup Shaders procedural data (Baradad et al.) visual memory StyleGAN-based procedural data
Closest papers by meaning
- Towards flexible perception with visual memory (ICML 2025 · similarity 0.88)
- VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning (ICLR 2025 · similarity 0.86)
- Towards Continual Universal Segmentation (CVPR 2025 · similarity 0.85)
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.85)
- DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks (NeurIPS 2025 · similarity 0.85)
- The Unreasonable Effectiveness of Linear Prediction as a Perceptual Metric (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.