Research map / Large language models / Language, safety and interpretability
Measuring Diversity in Synthetic Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DCScore, a classification-based diversity metric for synthetic datasets that treats each sample as its own class and sums the diagonal of a softmax probability matrix over kernel similarities, offering axiomatic guarantees and lower computational cost than VendiScore.
Problem
Measuring the diversity of LLM-generated synthetic datasets is under-explored; existing NLP/ML diversity metrics (n-gram, reference-based, eigenvalue-based transformation methods) are either form-only, need reference data, or are computationally expensive for large synthetic datasets.
Approach
Formulate diversity evaluation as an n-way sample classification problem: each sample's kernel similarities act as logits over per-sample classes, and the trace of the resulting softmax probability matrix (DCScore) serves as the diversity score, avoiding eigenvalue computation.
Key results
- On zero-shot text classification with Llama2-13B, DCScore achieves Spearman's rho 0.9961 with tau_g, matching VendiScore (0.9961) and above Distinct-n (0.9909) and K-means Inertia (-0.1143).
- Under few-shot setting, DCScore obtains the best or tied-best correlation with tau_g in all four columns of the table (e.g., 0.9974 for story completion with Llama2-70B).
- Computation time on SST2 at 4k samples is about 4.58s for DCScore vs 4.65s for VendiScore with inner product, and DCScore is slower at 64k (112.47s vs 76.19s) for inner product kernel.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DCScore instead of Distinct-n, K-means Inertia, or VendiScore | Synthetic text classification and story completion datasets · Few-shot τg correlation across datasets and generator sizes | Spearman’s ρ | Helped |
| Use DCScore instead of VendiScore | Synthetic datasets, τg correlation · Zero-shot settings | Spearman’s ρ | No clear effect |
| Measure diversity with DCScore rather than VendiScore | SST2 at 64k samples with inner-product kernel · SST2 | computation time: 76.19$_{\pm 1.91}$ -> 112.47$_{\pm 2.43}$ | Hurt |
| Measure diversity with DCScore rather than VendiScore | SST2 at 64k samples with RBF kernel · SST2 | computation time: 449.14$_{\pm 10.35}$ -> 117.06$_{\pm 1.91}$ | Helped |
| Generate the training data with higher τg (0.2 to 1.2) | RoBERTa trained on zero-shot synthetic text classification data · Zero-shot text classification | accuracy: 89.10 -> 90.37 | Helped |
| Generate the training data with higher τg (0.2 to 1.2) | RoBERTa trained on few-shot synthetic text classification data · Few-shot text classification | accuracy: 70.07 -> 73.41 | Helped |
| Increase the Softmax temperature τ (ablation) | DCScore · Self-generated text classification and story completion datasets | Spearman’s ρ with τg | Helped |
| Add duplicate samples to the evaluated dataset (ablation) | DCScore on SST2-AttrPrompt, Yelp-AttrPrompt, and AG News-AttrPrompt · Datasets with duplicate samples | effective sample count stability | No clear effect |
Surprising findings
- With the inner-product kernel and n much larger than d, VendiScore has lower asymptotic complexity than DCScore, and DCScore is slower at 64k samples in that setting.
- K-means Inertia shows near-zero correlation (-0.1143) with tau_g on zero-shot 13B text classification data.
Limitations
- Only evaluates unimodal data; cannot directly handle multimodal (image-text) datasets.
- Correlation with GPT-4 judgments is lower (about 0.6) in few-shot settings, and identical content in batches lowers human correlation for some settings.
- With inner-product kernel and n much larger than d, computational advantage over VendiScore disappears.
Experimental setup
Datasets: SST2 Yelp AG News AttrPrompt-augmented SST2/Yelp/AG News ROC Stories IMDB colored MNIST
Benchmarks: self-generated text classification and story completion datasets (tau_g sweep) SST2-AttrPrompt, Yelp-AttrPrompt, AG News-AttrPrompt colored MNIST
Metrics: Spearman's rho computation time effective number of samples
Baselines: Distinct-n K-means Inertia VendiScore RKE FKEA
Builds on: VendiScore Sentence-BERT AttrPrompt MAUVE
Closest papers by meaning
- Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data (NeurIPS 2025 · similarity 0.87)
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.86)
- DataComp-LM: In search of the next generation of training sets for language models (NeurIPS 2024 · similarity 0.86)
- SimpleStrat: Diversifying Language Model Generation with Stratification (NeurIPS 2025 · similarity 0.85)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.85)
- Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.