Research map / Large language models / Language, safety and interpretability
Parameterized Synthetic Text Generation with SimpleStories
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SimpleStories, a 2M-sample-per-language synthetic simple-story dataset (English and Japanese) generated with parameterized GPT-4o-mini prompts, plus a tiny model suite with a custom 4,096-token WordPiece tokenizer that is compared against TinyStories.
Problem
TinyStories is formulaic (59% of stories contain "Once upon a time"), unlabeled, English-only, not fully open-source, and contains artifacts and duplicates, which limits its use for supervised training and interpretability research on small models.
Approach
Generate simple stories with GPT-4o-mini using multi-level parameterized prompts (theme, topic, style, narrative feature, grammar feature, author persona, starting part-of-speech and letter constraints) so that stories are diverse and carry recoverable labels, and release all generation and training code.
Key results
- SimpleStories has lower compression ratio and Self-BLEU than TinyStories (significantly less repetitive), and higher n-gram diversity, with the most common 4-gram 'took a deep breath' at 7.49% vs 'once upon a time' at 59.38% in TinyStories.
- Judge recovery accuracy for labels is 0.49 for topic and 0.225, 0.145, 0.155 for theme, style and narrative feature.
- SimpleStories-35M with a custom tokenizer outperforms TinyStories-33M on coherence and quality in GPT-based judging, with the custom tokenizer adding +26.7 coherence and +27.5 quality points over the GPT-2 tokenizer on the same architecture.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the SimpleStories dataset instead of TinyStories | syntactic diversity of story datasets · English stories | template rate: 100 -> 88.9 | Helped |
| Use the SimpleStories dataset instead of TinyStories | syntactic diversity of story datasets · English stories | templates per token: 0.026 -> 0.016 | Helped |
| Use SimpleStories instead of TinyStories | story dataset · English stories | semantic content and style diversity | Helped |
| Use SimpleStories instead of TinyStories | story dataset · English stories | simplicity | No clear effect |
| Use SimpleStories instead of TinyStories | story dataset · English stories | compression ratio and Self-BLEU homogenization | Helped |
| Replace the GPT-2 tokenizer with the custom tokenizer (ablation) | SimpleStories-35M · generated stories | coherence: +26.7 points | Helped |
| Replace the GPT-2 tokenizer with the custom tokenizer (ablation) | SimpleStories-35M · generated stories | quality: +27.5 points | Helped |
| Replace the GPT-2 tokenizer with the custom tokenizer (ablation) | SimpleStories-35M · generated stories | originality, coherence, grammar, and quality | Helped |
| Use Llama architecture instead of the original architecture (ablation) | TinyStories-33M · generated stories | originality and coherence | Helped |
| Use Llama architecture instead of the original architecture (ablation) | TinyStories-33M · generated stories | four judge metrics versus SimpleStories-35M with custom tokenizer | Hurt |
| Use ordinary weighted pooling instead of average or softmax pooling (ablation) | story-label probes · probe classification | probe accuracy | Helped |
| Ablate attention (ablation) | trained 35M model probes · story-label probing | relative accuracy: 50%-80% of relative accuracy | Hurt |
Surprising findings
- The original TinyStories-33M nearly matches the best SimpleStories model on grammar despite lower scores on other metrics.
- Probe accuracy is notably lower near initial layers of the larger models, suggesting labels are represented internally rather than in output tokens.
Limitations
- No custom tokenizer was trained for TinyStories, and no BPE tokenizer comparison was done, so tokenizer effects are not fully isolated.
- Evaluation of the English dataset only; other languages are not fairly evaluated.
- Automated diversity and judge metrics are imperfect, and the lower bound on label quality is not fully elicited.
Experimental setup
Datasets: TinyStories SimpleStories
Benchmarks: SimpleStories TinyStories
Metrics: 4-gram frequency Flesch-Kincaid grade compression ratio Self-BLEU n-gram diversity score POS template rate template-per-token judge scores (originality, coherence, grammar, quality) probe accuracy
Baselines: TinyStories-33M TinyStories dataset
Builds on: TinyStories WordPiece Nucleus Sampling GPT-Eval
Closest papers by meaning
- Controlled Text Generation via Language Model Arithmetic (ICLR 2024 · similarity 0.86)
- Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs (ICLR 2025 · similarity 0.85)
- SimpleStrat: Diversifying Language Model Generation with Stratification (NeurIPS 2025 · similarity 0.85)
- The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation (ICLR 2025 · similarity 0.85)
- One-Prompt-One-Story: Free-Lunch Consistent Text-to-Image Generation Using a Single Prompt (ICLR 2025 · similarity 0.85)
- LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.