Research map / Label-efficient and robust learning / Recognition and long-tail
Testing Semantic Importance via Betting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes global, global-conditional, and local-conditional statistical importance of semantic concepts for black-box classifiers and tests them with sequential kernelized independence tests (SKIT, C-SKIT, X-SKIT) that give Type I/FDR control and a rank of importance.
Problem
Existing semantic/concept importance methods for black-box predictors lack statistical guarantees (false positive rate or FDR control), and cannot provide rigorous local and global significance for concepts.
Approach
Frame semantic importance as conditional independence between model predictions and concept projections, and test it via betting-based sequential kernel (MMD) tests with conditional randomization, yielding e-value-based rejection times that induce a rank of importance and FDR control via a greedy post-processor.
Key results
- SKIT achieves average rank agreement 0.50 and importance f1 0.65 on AwA2, outperforming PCBM across metrics.
- X-SKIT achieves rank agreement 0.82 across models and f1 0.93 against ground-truth attributes on CUB.
- On Imagenette, the tests retain classifier accuracy (98.99%) versus 95.85% for PCBM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SKIT instead of PCBM for global semantic importance ranking | Imagenette · Imagenette | rank agreement: 0.45 -> 0.51 | Helped |
| Use SKIT instead of PCBM for the classifier | Imagenette · Imagenette | accuracy: 95.85% -> 98.99% | Helped |
| Use SKIT instead of PCBM for global semantic importance ranking | AwA2 · AwA2 | rank agreement: 0.36 -> 0.50 | Helped |
| Use SKIT instead of PCBM for global semantic importance ranking | AwA2 · AwA2 | importance F1: 0.53 -> 0.65 | Helped |
| Use C-SKIT instead of PCBM for global conditional semantic importance ranking | AwA2 · AwA2 | rank agreement: 0.36 -> 0.46 | Helped |
| Use C-SKIT instead of PCBM for global conditional semantic importance ranking | AwA2 · AwA2 | importance F1: 0.53 -> 0.57 | Helped |
| Use X-SKIT for local semantic importance ranking | zero-shot image classification models · CUB | rank agreement: 0.82 | Helped |
| Use X-SKIT for local semantic importance ranking | zero-shot image classification models · CUB | importance F1: 0.93 | Helped |
| Use global conditional testing rather than marginal testing | C-SKIT versus SKIT · AwA2 | rejection rate | Hurt |
Surprising findings
- Linear kernel fails to reject in the C-SKIT synthetic experiment, showing kernels must be characteristic for power.
Limitations
- Requires samplers for conditional distributions of concepts or embeddings, which may be hard to learn.
- Kernel tests need characteristic kernels, which may fail for discrete data or graphs.
- No guarantee that user-specified concepts are well represented in the model's feature space.
Experimental setup
Datasets: AwA2 CUB-200-2011 Imagenette
Benchmarks: AwA2 CUB-200-2011 Imagenette
Metrics: rejection rate rejection time rank agreement accuracy f1
Baselines: PCBM SpLiCe
Builds on: SKIT (Podkopaev et al. / Shekhar and Ramdas) conditional randomization test (Candes et al.) explanation randomization test (XRT) post-hoc concept bottleneck models SpLiCe
Closest papers by meaning
- Conditional Feature Importance with Generative Modeling Using Adversarial Random Forests (AAAI 2025 · similarity 0.86)
- Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models (ICML 2026 · similarity 0.85)
- Nonparametric Identification of Latent Concepts (ICML 2025 · similarity 0.84)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.84)
- Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering (CVPR 2024 · similarity 0.83)
- Variable Importance in High-Dimensional Settings Requires Grouping (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.