Research map / Vision-language and multimodal / Human and embodied modeling
Explaining Datasets in Words: Statistical Models with Natural Language Parameters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces statistical models (clustering, time series, classification) whose parameters are natural language predicates, learned by gradient-optimizing continuous relaxations and discretizing them with LLM prompting.
Problem
High-dimensional parameters of classical dataset-explanation models (cluster centers, topic word weights, n-gram regression) are hard to interpret, so explanations of datasets are not directly human-readable.
Approach
Parameterize clustering, time series, and classification models with natural language predicates whose 0/1 denotations are computed by an LLM; learn them by optimizing continuous relaxations (unit vectors over text embeddings) with gradient descent, then discretize into predicates via LLM prompting, with iterative refinement of the least useful predicate.
Key results
- On clustering over five datasets, the method averages F1 0.64 and surface 0.57, versus 0.62 F1 for GoalEx and 0.56 F1 for No-Relax.
- On time series and classification, the full method reaches 0.73 F1 on classification versus 0.58 for No-Refine and 0.68 for No-Relax.
- Applied to LLM usage, the classification model finds Llama-3-70B is better on open-ended questions and worse on technical questions or code snippets relative to Claude-3-Opus.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use iterative refinement instead of stopping after initial discretization (ablation) | Natural-language predicate learning · Clustering benchmarks | F1: 0.53 -> 0.64 | Helped |
| use continuous relaxation to guide predicate selection instead of selecting predicates randomly (ablation) | Natural-language predicate learning · Clustering benchmarks | F1: 0.56 -> 0.64 | Helped |
| use continuous relaxation to guide predicate selection instead of selecting predicates randomly (ablation) | Natural-language predicate learning · topic, lang, locat, all, time-avg, classification | F1: topic: 0.65 -> 0.67; lang: 0.52 -> 0.62; locat: 0.48 -> 0.55; all: 0.61 -> 0.72; time-avg: 0.56 -> 0.64; classification: | Helped |
| use neural embeddings instead of one-hot text encoding (ablation) | Natural-language predicate learning · Clustering benchmarks | F1: 0.50 -> 0.64 | Helped |
| use the model-agnostic method rather than GoalEx | Clustering · AGNews, NYT, Bills, Wiki | F1: AGNews: 0.86 vs 0.86; NYT: 0.70 vs 0.68; Bills: 0.45 vs 0.33; Wiki: 0.51 vs 0.49 | Mixed |
| use the model-agnostic method rather than GoalEx | Clustering · DBPedia | F1: 0.75 -> 0.68 | Hurt |
| use natural-language predicate clustering rather than K-means | Clustering · Clustering average | F1: 0.65 -> 0.64 | Hurt |
| use natural-language predicate clustering rather than topic modeling | Clustering · Clustering average | F1: 0.43 -> 0.64 | Helped |
| use neural embeddings rather than one-hot text encoding (ablation) | Natural-language predicate learning · topic, lang, locat, all, time-avg, classification | F1: topic: 0.63 -> 0.67; lang: 0.51 -> 0.62; locat: 0.51 -> 0.55; all: 0.66 -> 0.72; time-avg: 0.58 -> 0.64; classification: | Mixed |
Surprising findings
- Naive prompting for predicates (no relaxation or refinement) performs much worse, averaging 0.27 F1 on clustering.
- The method's average clustering F1 (0.64) is close to K-means (0.65) and clearly above topic modeling (0.43).
Limitations
- Slow: denotation computation requires many LLM API calls.
- Predicates are often semantically redundant and add little information.
- Hard to control predicates to satisfy structural constraints such as subset relations between parent and child categories.
Experimental setup
Datasets: AG-News DBPedia NYT Bills Wiki MATH WildChat LaMem Translated NYT (Spanish, French, German)
Benchmarks: AG-News DBPedia NYT Bills Wiki NYT-based time series (all, topic, locat, lang) NYT-based multiclass classification
Metrics: F1 similarity surface form similarity (LLM-judged 1/0.5/0)
Baselines: GoalEx K-means TopicModel (LDA) Prompting No-Relax No-Refine OneHot embeddings
Builds on: GoalEx Concept Bottleneck Models VisDiff Instructor embeddings
Closest papers by meaning
- Automated Statistical Model Discovery with Language Models (ICML 2024 · similarity 0.86)
- LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language (NeurIPS 2024 · similarity 0.84)
- Exploring Activation Patterns of Parameters in Language Models (AAAI 2025 · similarity 0.84)
- Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention (AAAI 2024 · similarity 0.84)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.84)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.