Research map / Label-efficient and robust learning / Recognition and long-tail
Inconsistency-Based Data-Centric Active Open-Set Annotation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NEAT, a data-centric active open-set annotation method that uses CLIP features and K-NN label clusterability to find known-class samples, then selects samples by a cross-entropy inconsistency between model predictions and local neighbor label distribution.
Problem
Active learning with open-set unlabeled pools, where unknown-class samples must be excluded and informative known-class samples selected under a labeling budget.
Approach
Replaces a learned detector network (LfOSA) with a training-free data-centric known-class detector based on K-NN label clusterability over CLIP features, and scores selected samples by the cross-entropy between the classifier's prediction and the neighbors' label distribution (inconsistency), approximating version-space disagreement without training multiple models.
Key results
- NEAT improves accuracy over LfOSA by 6% on CIFAR10, 11% on CIFAR100, and 10% on Tiny-ImageNet (reported in the paper).
- NEAT recall improves over LfOSA by 4%, 12%, and 7% on CIFAR10, CIFAR100, and Tiny-ImageNet respectively.
- Training time on Tiny-ImageNet is 88 minutes for NEAT versus 156 minutes for LfOSA on an NVIDIA A5000.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NEAT instead of LfOSA | Active open-set annotation · CIFAR10 | accuracy: 6% | Helped |
| use NEAT instead of LfOSA | Active open-set annotation · CIFAR100 | accuracy: 11% | Helped |
| use NEAT instead of LfOSA | Active open-set annotation · Tiny-ImageNet | accuracy: 10% | Helped |
| use NEAT instead of LfOSA | Active open-set annotation · CIFAR10 | recall: 4% | Helped |
| use NEAT instead of LfOSA | Active open-set annotation · CIFAR100 | recall: 12% | Helped |
| use NEAT instead of LfOSA | Active open-set annotation · Tiny-ImageNet | recall: 7% | Helped |
| use NEAT instead of LfOSA | Active open-set annotation · CIFAR10, CIFAR100, Tiny-ImageNet | training time: 88 minutes vs 156 minutes; 21 minutes vs 59 minutes; 17 minutes vs 25 minutes | Helped |
| use CLIP instead of ImageNet-pretrained ResNet features (ablation) | NEAT · CIFAR10 and CIFAR100 | accuracy: 97.48 ± 0.26 -> 98.15 ± 0.14; 60.75 ± 0.58 -> 63.68 ± 0.67 | Helped |
| use CLIP instead of ImageNet-pretrained ResNet features (ablation) | NEAT · Tiny-ImageNet | accuracy: 50.53 ± 1.15 -> 49.01 ± 1.60 | Hurt |
| set number of neighbors K to 5 instead of 10 (ablation) | NEAT · Tiny-ImageNet | accuracy: 49.01 ± 1.60 -> 47.23 ± 1.21 | Hurt |
| vary number of neighbors K from 10 to 15 or 20 (ablation) | NEAT · Tiny-ImageNet | accuracy: 49.01 ± 1.60 -> 49.02 ± 0.61; 49.01 ± 1.60 -> 50.73 ± 0.29 | Helped |
| vary number of neighbors K from 10 to 5, 15, or 20 (ablation) | NEAT · CIFAR100 | accuracy: 63.68 ± 0.67 -> 62.63 ± 0.66; 62.77 ± 0.84; 60.85 ± 1.70 | Mixed |
Surprising findings
- NEAT (Passive), which randomly samples from detected known classes, achieves slightly higher recall and precision than full NEAT, though NEAT has higher accuracy.
- Replacing ResNet features with CLIP features did not help on Tiny-ImageNet (CLIP 49.01 vs ResNet-50 50.53), though CLIP was better on CIFAR10 and CIFAR100.
Limitations
- Known-class detection relies on feature quality; the theoretical bound depends on assumptions about label clusterability and labeling error.
- Evaluated only on small image classification benchmarks with a limited number of known classes (2, 20, 40).
- Using pre-trained features introduces prior knowledge that may not be fair to other active learning baselines.
Experimental setup
Datasets: CIFAR10 CIFAR100 Tiny-ImageNet
Benchmarks: CIFAR10 CIFAR100 Tiny-ImageNet
Metrics: accuracy precision recall
Baselines: Random Uncertainty Certainty Coreset BGADL OpenMax BADGE MQNet LfOSA
Builds on: LfOSA version-space active learning CLIP OpenMax
Closest papers by meaning
- Rethinking Epistemic and Aleatoric Uncertainty for Active Open-Set Annotation: An Energy-Based Approach (CVPR 2025 · similarity 0.88)
- Joint Out-of-Distribution Filtering and Data Discovery Active Learning (CVPR 2025 · similarity 0.85)
- Improved Algorithm for Deep Active Learning under Imbalance via Optimal Separation (ICML 2025 · similarity 0.85)
- Entropic Open-Set Active Learning (AAAI 2024 · similarity 0.85)
- Towards Cost-Effective Learning: A Synergy of Semi-Supervised and Active Learning (CVPR 2025 · similarity 0.84)
- Instance-wise Supervision-level Optimization in Active Learning (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.