Research map / Label-efficient and robust learning / Recognition and long-tail
Understanding Distributed Representations of Concepts in Deep Neural Networks without Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Relaxed Decision Regions (RDR), an unsupervised method that uses a Hamming-based configuration distance over ReLU activation states and greedily selects principal neurons to group instances sharing coherent learned concepts.
Problem
Interpreting distributed representations of concepts learned by DNN classifiers without human supervision (no pre-defined concept sets or segmentation labels), including discovering unlabeled subclasses and explaining misclassifications.
Approach
Defines a configuration distance (Hamming distance between ReLU activation-state vectors) to find instances sharing concepts, then greedily selects the principal subset of neurons whose activation states best separate the positive neighbor set from negatives, forming an interpretable region (RDR) in feature space.
Key results
- On Oxford-Pet subclass grouping, RDR achieves purity 0.351 (VGG19), 0.408 (ResNet50), 0.346 (MobileNetV2), outperforming KNN_C (0.303/0.328/0.329) and RI (0.045/0.056/0.056).
- In a 22-participant human study, 81.8% of participants identified the target's RDR group among five same-class samples, and 85.2% judged RDR samples clearer than other grouping methods.
- The method runs in under 2 seconds after a roughly 15-second forward pass for VGG19 on Mini-ImageNet on a single GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RDR instead of CAV_C | Oxford-Pet subclass grouping with VGG19 · Oxford-Pet | purity: 0.314 -> 0.351 | Helped |
| use RDR instead of CAV_C | Oxford-Pet subclass grouping with VGG19 · Oxford-Pet | entropy: 1.549 -> 1.527 | Helped |
| use RDR instead of CAV_C | Oxford-Pet subclass grouping with ResNet50 · Oxford-Pet | purity: 0.387 -> 0.408 | Helped |
| use RDR instead of CAV_C | Oxford-Pet subclass grouping with ResNet50 · Oxford-Pet | entropy: 1.416 -> 1.372 | Helped |
| use RDR instead of KNN_C | Oxford-Pet subclass grouping with MobileNetV2 · Oxford-Pet | purity: 0.329 -> 0.346 | Helped |
| use RDR instead of KNN_C | Oxford-Pet subclass grouping with MobileNetV2 · Oxford-Pet | entropy: 1.498 -> 1.531 | No clear effect |
| use Configuration distance instead of Euclidean distance (ablation) | RDR on Oxford-Pet · Oxford-Pet | purity: 0.241 -> 0.351; 0.252 -> 0.408; 0.303 -> 0.346 | Helped |
| use Configuration distance instead of Euclidean distance (ablation) | RDR on Oxford-Pet · Oxford-Pet | entropy: 1.76 -> 1.527; 1.779 -> 1.372; 1.76 -> 1.531 | Helped |
| use Configuration distance instead of Cosine distance (ablation) | RDR on Oxford-Pet · Oxford-Pet | purity: 0.309 -> 0.351; 0.307 -> 0.408 | Helped |
| use Configuration distance instead of Cosine distance (ablation) | RDR on Oxford-Pet · Oxford-Pet | entropy: 1.613 -> 1.527; 1.7 -> 1.372; 1.628 -> 1.531 | Mixed |
Surprising findings
- Cosine-similarity-based neighbors tend to have small configuration distances, offering an explanation for cosine's empirical success in feature spaces.
- Representative Interpretation and CAR produce overly broad regions that merge distinct subclasses.
Limitations
- Relies on the parameters t and k, which need tuning per layer due to feature sparsity differences.
- Applies only to piecewise linear (ReLU-family) networks; the theory and method assume this structure.
- Many qualitative results rest on visual inspection; quantitative evaluation is limited to Oxford-Pet subclasses and a single human study.
Experimental setup
Datasets: Mini-ImageNet Flowers Recognition CIFAR-10 Oxford-Pet Broden ImageNet-X
Benchmarks: Oxford-Pet subclass purity/entropy Broden concept alignment ImageNet-X failure types
Metrics: purity entropy human agreement rate
Baselines: K-Nearest Neighbors Representative Interpretation (RI) CAR CAV Grad-CAM Integrated Gradients ACE
Builds on: Representative Interpretation Concept Activation Vectors (TCAV) Concept Activation Regions Automated Concept-based Explanation (ACE) Grad-CAM
Closest papers by meaning
- FaCT: Faithful Concept Traces for Explaining Neural Network Decisions (NeurIPS 2025 · similarity 0.85)
- Knowledge-Aware Neuron Interpretation for Scene Classification (AAAI 2024 · similarity 0.85)
- Automated Natural Language Explanation of Deep Visual Neurons with Large Models (Student Abstract) (AAAI 2024 · similarity 0.85)
- Finding NEM-U: Explaining unsupervised representation learning through neural network generated explanation masks (ICML 2024 · similarity 0.85)
- Show and Tell: Visually Explainable Deep Neural Nets via Spatially-Aware Concept Bottleneck Models (CVPR 2025 · similarity 0.85)
- Representational Similarity via Interpretable Visual Concepts (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.