Research map / Large language models / Agents, code and math
SS-GEN: A Social Story Generation Framework with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SS-GEN, a constraint-driven hierarchical LLM data-synthesis pipeline (StarSow) that builds a ~5K Social Story dataset with GPT-4o, plus quality criteria, and fine-tunes small open LLMs (Gemma, Mistral, LLaMA3) on it with LoRA.
Problem
Social Stories for autistic children are expensive to write by hand and lack diversity, and general LLMs struggle to meet their strict structural, descriptive and safety constraints.
Approach
A breadth-first, hierarchical synthesis pipeline (StarSow: chapters -> titles -> story content, with filtering) guided by constraint-driven prompts to generate Social Stories with GPT-4o, paired with quality criteria and used to fine-tune smaller models cheaply.
Key results
- Fine-tuning Gemma (2B) on the dataset raised BLEU-4 from 8.86 to 51.18 and ROUGE-L from 7.22 to 27.29 in zero-shot vs SFT comparison.
- Human evaluation: zero-shot Gemma-1.1-Instruct (2B) met the situational safety criteria in only 6% of stories, versus 80% for SFT Gemma (2B).
- In the GPT-3 vs GPT-4 quality check, GPT-4 was preferred 62% of the time over GPT-3 for regenerated stories, and descriptive orientation was only 56-59%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune Gemma (2B) on the SS-GEN dataset | Gemma (2B) · SS-GEN test set | BLEU-4: 8.86 -> 51.18 | Helped |
| fine-tune models on the SS-GEN dataset | Gemma, Mistral, and Llama3 models · SS-GEN test set | traditional metrics | Helped |
| use an Instruct model instead of its base pretrained counterpart | Zero-Shot SS-GEN models · SS-GEN test set | traditional metrics: 33.57%↑ (BLEU-4), 13.17%↑ (ROUGE-1), 4.19%↑ (ROUGE-2), and 10%↑ (ROUGE-L) | Helped |
| fine-tune a base model rather than an Instruct model | Gemma and Mistral · SS-GEN test set | traditional metrics | Helped |
| fine-tune a base model rather than an Instruct model | Llama3 · SS-GEN test set | traditional metrics | Mixed |
| fine-tune Gemma (7B) on the SS-GEN dataset instead of using Gemma-1.1-Instruct (7B) Zero-Shot | Gemma (7B) · SS-GEN human evaluation | Situational Safety: 10.0% -> 83.3% | Helped |
Surprising findings
- Base models outperformed their Instruct counterparts after SFT for Gemma and Mistral, but not for LLaMA3.
- Generated stories scored only 50-60% on the descriptive orientation criterion despite high scores on other constraints.
Limitations
- Only 74.3% of generated data survived filtering, requiring manual screening.
- Human evaluation involved only three psychology students and small samples (50 stories for experts).
- Ethical risks include training-data bias and need for caregiver supervision of AI-drafted stories.
Experimental setup
Datasets: Social Story dataset constructed via StarSow from a 179-story expert seed set
Benchmarks: SS-GEN Social Story test split
Metrics: BLEU-4 ROUGE-1 ROUGE-2 ROUGE-L BertScore Coherence Descriptiveness Empathy Grammaticality Relevance
Baselines: GPT-3.5-turbo Gemma-1.1-Instruct Mistral-v0.1-Instruct LLaMA3-Instruct
Builds on: LLaMA Factory LoRA
Closest papers by meaning
- SocialSim: Towards Socialized Simulation of Emotional Support Conversation (AAAI 2025 · similarity 0.84)
- Position: LLM Social Simulations Are a Promising Research Method (ICML 2025 · similarity 0.84)
- Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) (NeurIPS 2025 · similarity 0.82)
- Explore Theory of Mind: program-guided adversarial data generation for theory of mind reasoning (ICLR 2025 · similarity 0.82)
- SNS-Bench: Defining, Building, and Assessing Capabilities of Large Language Models in Social Networking Services (ICML 2025 · similarity 0.82)
- SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.