Research map / Detection and segmentation / Open-set and few-shot detection

TDDBench: A Benchmark for Training data detection

ICLR 2025 · Detection and segmentation › Open-set and few-shot detection · approach: Industrial anomaly detection · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces TDDBench, an open-source benchmark evaluating 21 training data detection (membership inference) algorithms across 13 image, tabular, and text datasets and 41 target models, including Pythia-12B on WikiMIA.

Problem

Existing training data detection (membership inference) benchmarks cover few modalities and algorithms, rarely include recent deep learning and LLM methods, ignore target model effects, and overlook efficiency and memory costs.

Approach

A multi-modality, multi-algorithm benchmark that groups 21 TDD methods into metric-based, learning-based, model-based, and query-based families and evaluates detection performance alongside runtime and memory, target model size, train-test gap, limited architecture knowledge, and defenses.

Key results

• Model-based methods achieve the highest average AUROC (0.706 average, 0.728 best) across 21 algorithms in the efficiency-tradeoff table, while most methods land between 0.5 and 0.6 AUC in most settings.
• Query-ref reaches AUROC 0.941 on CIFAR-100 and 0.735 on CIFAR-10 but is only applicable to image data, and it trains reference models.
• Detection performance is positively correlated with the target model's train-test accuracy gap, and it generally improves with model size (e.g., Pythia on WikiMIA).

Measured effects reported in the paper

Surprising findings

• Overall AUC for most TDD algorithms falls between 0.5 and 0.6, indicating unsatisfactory detection across datasets.
• Introducing small amounts of noise to the target model's output substantially degrades learning-based and model-based detectors.

Limitations

• Evaluation of most TDD algorithms is restricted to the black-box setting with fixed training protocols.
• Experiments on LLMs are limited to Pythia and WikiMIA, with many methods not applied to large models due to compute constraints.
• Query-ref is only implemented for image data, so the text and tabular comparison omits it.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 BloodMNIST CelebA Purchase Texas Adult Student Rotten Tomatoes Tweet Eval GLUE-CoLA ECtHR Articles

Benchmarks: TDDBench WIKIMIA CIFAR-10 CIFAR-100 BloodMNIST CelebA Purchase Texas Adult Student Rotten Tomatoes Tweet Eval

Metrics: AUROC Precision Accuracy TPR@1% FPR TPR@10% FPR F1-score Recall FNR FPR membership advantage

Baselines: Metric-loss Metric-conf Metric-corr Metric-ent Metric-ment Learn-original Learn-top3 Learn-sorted Learn-label Learn-merge Model-loss Model-calibration

Builds on: shadow training (membership inference attacks) LiRA Min-K%

Closest papers by meaning

• Membership Inference Attacks against Large Vision-Language Models (NeurIPS 2024 · similarity 0.85)
• quanda: An Interpretability Toolkit for Training Data Attribution Evaluation (NeurIPS 2026 · similarity 0.85)
• MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector (AAAI 2025 · similarity 0.84)
• Causal Evaluation of Membership Inference Attacks (NeurIPS 2026 · similarity 0.84)
• LLM Dataset Inference: Did you train on my dataset? (NeurIPS 2024 · similarity 0.83)
• idSCD: Identifying Training Datasets through Semantic Correlation Descriptors (NeurIPS 2026 · similarity 0.83)