Research map / Large language models / Language, safety and interpretability
Causally Evaluating the Learnability of Formal Language Tasks
Vésteinn Snæbjarnarson, Anej Svete, Josef Valvoda, Reda Boumasmoud, Brian DuSell, Ryan Cotterell
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use causal intervention (exact-count sampling) instead of correlational sampling to measure learnability | LSTM and transformer trained on PFSA corpora · Parity + star-free PFSA | decomposed KL | Mixed |
| use causal instead of correlational sampling over 50-state random PFSAs (symbol-level) | LSTM and transformer · Random 50-state 10-symbol PFSAs | decomposed KL: correlational KL first rises to a peak near 250 occurrences before falling | Mixed |
| use causal instead of correlational sampling on a fixed 40-state topology with varied weights | LSTM and transformer · Fixed 40-state PFSA, state-level | decomposed KL | Mixed |
| increase parity-state occurrence count in training corpus for LSTM versus transformer | LSTM vs transformer · Parity + star-free PFSA, odd state | per-state KL | Mixed |
Closest papers by meaning
- Can Large Language Models Infer Causation from Correlation? (ICLR 2024 · similarity 0.87)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.87)
- Compositional Causal Reasoning Evaluation in Language Models (ICML 2025 · similarity 0.87)
- Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective (ICML 2024 · similarity 0.86)
- Automated Assessment of Fidelity and Interpretability: An Evaluation Framework for Large Language Models’ Explanations (Student Abstract) (AAAI 2024 · similarity 0.86)
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
