Research map / Architectures and efficiency / Attention and transformers
Context-free Recognition with Transformers
Selim Jerad, Anej Svete, Sophie Hao, Ryan Cotterell, William Merrill
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add O(log N) looping (ablation) | transformer classifier · Balanced Counting | max accuracy: 90 -> 94 | Improved |
| add O(log N) looping (ablation) | transformer classifier · Dyck(1) | max accuracy: 85 -> 86 | No clear effect |
| add O(log N) looping and O(N) padding (ablation) | transformer classifier · Dyck(2) | max accuracy: 83 -> 87 | Improved |
| add O(log N) looping and O(N) padding (ablation) | transformer classifier · Palindrome | max accuracy: 68 -> 72 | Improved |
| add O(log N) looping only (ablation) | transformer classifier · Palindrome | max accuracy: 68 -> 67 | No clear effect |
| add O(log N) looping only (ablation) | transformer classifier · BFVP (infix) | max accuracy: 80 -> 78 | Hurt |
| add O(log N) looping and O(N) padding (ablation) | transformer classifier · BFVP (infix) | max accuracy: 80 -> 81 | No clear effect |
| add O(log N) looping (ablation) | transformer classifier · BFVP (postfix) | max accuracy: 67 -> 75 | Improved |
| add O(N) padding on top of looping (ablation) | transformer classifier · BFVP (postfix) | max accuracy: 75 -> 75 | No clear effect |
Closest papers by meaning
- Exact Expressive Power of Transformers with Padding (NeurIPS 2025 · similarity 0.86)
- A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers (NeurIPS 2025 · similarity 0.86)
- Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers (NeurIPS 2025 · similarity 0.86)
- Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns (ICLR 2024 · similarity 0.85)
- How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations (ICLR 2024 · similarity 0.85)
- Training Neural Networks as Recognizers of Formal Languages (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
