Research map / Probabilistic and scientific modeling / Causal discovery

idSCD: Identifying Training Datasets through Semantic Correlation Descriptors

NeurIPS 2026 · Probabilistic and scientific modeling › Causal discovery · approach: Causal representation identifiability · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes semantic correlation descriptors (SCDs) built from keyword-class correlation maps (BEE on Pythia-14M) to perform white-box dataset-level membership inference, outperforming RMIA, Attack-P, LiRA and SIF on NLI, emotion and medical groups.

Problem

Determining whether a whole dataset was part of a model's training mixture, which existing output-based (confidence, loss, generated samples) and influence-based methods address only indirectly.

Approach

Treat the keyword-class correlation map of a trained model (BEE scores over a shared YAKE keyword vocabulary with zero-fill alignment) as a dataset fingerprint; test membership by Pearson correlation between the inspected model's descriptor and a target dataset's standalone descriptor.

Key results

• id_SCD achieves ROC-AUC 0.980 on emotion, about 8% above the strongest baseline.
• On the medical group id_SCD improves over all baselines by more than 60% in ROC-AUC.
• On NLI id_SCD is about 3% below the best baseline (ROC-AUC 0.9105), attributed to lower keyword overlap (about 15%).

Surprising findings

• Lower shared keyword support (NLI, about 15% overlap) substantially weakens the SCD signal, so the method is not uniformly best.
• Output-based baselines fail to capture high-level semantic differences between medical topic clusters.

Limitations

• Requires white-box access to class weights and embeddings; black-box variants are only speculative.
• Dependence on shared keyword support; performance drops when keyword overlap is sparse (NLI).
• Operates at dataset-descriptor level and may confuse closely related datasets; controlled leave-one-out analysis is costly as dataset count grows.

Experimental setup

Datasets: EmoLit EmotionDataset20 GoEmotions XED SMED Reddit-L2 EFCAMDAT iTalki PubMed 200k RCT

Benchmarks: NLI dataset group emotion dataset group medical dataset group (PubMed 200k RCT clusters)

Metrics: ROC-AUC standard deviation across groups

Baselines: RMIA Attack-P LiRA SIF

Builds on: BEE YAKE LiRA RMIA SIF

Closest papers by meaning

• quanda: An Interpretability Toolkit for Training Data Attribution Evaluation (NeurIPS 2026 · similarity 0.84)
• Can Large Language Models Infer Causation from Correlation? (ICLR 2024 · similarity 0.84)
• Bayesian Inference for Correlated Human Experts and Classifiers (ICML 2025 · similarity 0.83)
• LLM Dataset Inference: Did you train on my dataset? (NeurIPS 2024 · similarity 0.83)
• Causal Representation Learning from Multimodal Biomedical Observations (ICLR 2025 · similarity 0.83)
• TDDBench: A Benchmark for Training data detection (ICLR 2025 · similarity 0.83)