Research map / Federated learning and privacy / Differential privacy
Struct-Bench: A Benchmark for Differentially Private Structured Text Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Struct-Bench, a CFG-based framework and benchmark (5 real + 2 synthetic structured-text datasets) for evaluating DP synthetic structured text, benchmarking PE, DP fine-tuning and baselines, plus a PE improvement case study.
Problem
Existing synthetic data metrics (FID, precision/recall, tabular marginals) fail to capture structural constraints and correlations in structured datasets containing natural language fields, so DP synthetic data generators cannot be properly compared on such data.
Approach
Represent each dataset's structure as a context-free grammar and parse samples into nodes, then compute structural metrics (CFG pass rate, key node dependency, attribute match) alongside KNN precision/recall and downstream TSTR accuracy.
Key results
- On ShareGPT at epsilon=4, PE with GPT-4o reaches CFG-PR 0.86 but KNN-Recall only 0.15, while DP-FT with GPT-2 gets CFG-PR 0 across datasets.
- On ICLR, all DP baselines achieve CFG-PR below 0.2 at epsilon=4.
- The case study combining CFG reformatting and node extraction raises PE's CFG-PR on ShareGPT (Llama3-8B) to about 94%, while improving KNN metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use instruction-guided conditional generation instead of unconditional DP fine-tuning | GPT-2 DP-FT on ShareGPT · ShareGPT | CFG-PR: 0 -> 0.55 | Helped |
| Use instruction-guided conditional generation instead of unconditional DP fine-tuning | GPT-2 DP-FT on ICLR · ICLR | CFG-PR: 0 -> 0.08 | Helped |
| Use GPT-4o instead of GPT-2 for PE | PE on ShareGPT · ShareGPT | CFG-PR: 0.8633 -> 0.8700 | Helped |
| Use GPT-4o instead of GPT-2 for PE | PE on ShareGPT · ShareGPT | KNN-Recall: 0.1528 -> 0.2627 | Hurt |
| Use LLM-assisted CFG reformatting (ablation) | PE on ShareGPT, ε=4 · ShareGPT | CFG-PR: + over 20% | Helped |
| Reformat samples after private voting rather than before voting (ablation) | PE with CFG reformat on ShareGPT, ε=4 · ShareGPT | CFG-PR | Helped |
| Reformat voted samples after voting (ablation) | PE on ShareGPT, ε=4 · ShareGPT | semantic and statistic properties | Hurt |
| Extract queries and auto-generate responses in the Variation API (ablation) | PE on ShareGPT · ShareGPT | KNN-Recall | Helped |
| Extract responses and auto-generate queries instead of extracting queries (ablation) | PE on ShareGPT · ShareGPT | KNN-Recall | Hurt |
| Extract key nodes during variation and auto-generate remaining nodes (ablation) | PE on ShareGPT · ShareGPT | CFG-PR | Helped |
| Combine CFG reformatting with query extraction and response auto-generation (ablation) | PE on ShareGPT, ε=4 · ShareGPT | CFG-PR: up to 94% | Helped |
| Increase ε for vanilla PE (ablation) | PE on ShareGPT · ShareGPT | KNN-Recall: around 0.35 | No clear effect |
Surprising findings
- DP fine-tuning reaches CFG-PR 0 at epsilon=4 on all datasets, and even non-private fine-tuning learns structure only on ShareGPT.
- Synthetic samples that violate structural constraints can still score high on KNN precision, showing semantic metrics alone miss structure.
Limitations
- CFG specification requires manual effort per dataset and domain knowledge.
- Key node dependency only considers pairwise dependencies, not higher-order relations.
- Semantic dependencies cannot be enforced as hard constraints since CFGs are context-free.
Experimental setup
Datasets: ShareGPT ICLR 2024 reviews Water Arena Adult Synthetic Reviews Synthetic Grounding
Benchmarks: Struct-Bench
Metrics: CFG-PR Key Node Dependency Attribute Match KNN-Precision KNN-Recall downstream accuracy (TSTR)
Baselines: Instruction Following (IF) DP fine-tuning (DP-FT) Instruct DP-FT Real data fine-tuning (FT) Private Evolution (PE)
Builds on: Private Evolution Augmented Private Evolution DP-SGD GEM benchmark Tregex
Closest papers by meaning
- SynBench: A Benchmark for Differentially Private Text Generation (NeurIPS 2026 · similarity 0.93)
- Synthesizing Privacy-Preserving Text Data via Finetuning *without* Finetuning Billion-Scale LLMs (ICML 2025 · similarity 0.88)
- Differentially Private Synthetic Data via Foundation Model APIs 2: Text (ICML 2024 · similarity 0.88)
- PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs (ICML 2024 · similarity 0.88)
- Private Evolution Converges (NeurIPS 2025 · similarity 0.87)
- A Privacy-Preserving Framework for Generative Model-driven Synthetic Datasets (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.