Research map / Large language models / Agents, code and math
AST-T5: Structure-Aware Pretraining for Code Generation and Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AST-T5, an encoder-decoder pretrained with AST-aware segmentation and AST-aware subtree span corruption, achieving strong results on code generation, transpilation and understanding at 277M parameters.
Problem
Code LMs treat code as token sequences, ignoring its syntactic structure; existing structure-aware methods need costly program analysis or execution, and greedy segmentation fragments AST structures during pretraining.
Approach
Use Tree-sitter ASTs with a dynamic-programming segmentation that minimizes AST-subtree breaks across chunk boundaries, and a span-corruption objective that masks whole AST subtrees (granularity controlled by a random threshold theta), with no architecture changes relative to T5.
Key results
- AST-T5 (277M) reaches 14.0 Pass@1 on HumanEval and 22.9 EM on Concode, outperforming CodeT5-Base baselines in the Base-scale comparison.
- Improves Bugs2Fix EM to 23.8 (small) versus CodeT5 21.6, and Java-C# transpilation to 68.9/72.3 versus CodeT5 65.9/66.9.
- Ablation: AST-aware segmentation and subtree corruption each add gains over a T5 baseline (average 44.2 to 47.9 with 25% mask ratio); 50% masking gave only marginal HumanEval gain and hurt other tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AST-Aware Segmentation instead of Greedy Segmentation (ablation) | T5 code pretraining · HumanEval | Pass@1: 5.2 -> 7.2 | Helped |
| add AST-Aware Segmentation instead of Greedy Segmentation (ablation) | T5 code pretraining · Concode | EM: 18.3 -> 20.2 | Helped |
| add AST-Aware Segmentation instead of Greedy Segmentation (ablation) | T5 code pretraining · All evaluation tasks (Avg) | average performance: 44.2 -> 45.7 | Helped |
| replace Vanilla T5 Span Corruption with AST-Aware Subtree Corruption (ablation) | T5 with AST-Aware Segmentation · HumanEval | Pass@1: 7.2 -> 9.6 | Helped |
| replace Vanilla T5 Span Corruption with AST-Aware Subtree Corruption (ablation) | T5 with AST-Aware Segmentation · All evaluation tasks (Avg) | average performance: 45.7 -> 47.0 | Helped |
| increase masking ratio from 15% to 25% (ablation) | AST-T5 with AST-aware pretraining · HumanEval | Pass@1: 9.6 -> 14.0 | Helped |
| increase masking ratio from 15% to 25% (ablation) | AST-T5 with AST-aware pretraining · All evaluation tasks (Avg) | average performance: 47.0 -> 47.9 | Helped |
| increase masking ratio from 15% to 25% (ablation) | AST-T5 with AST-aware pretraining · Defect Detect | accuracy: 66.0 -> 65.8 | No clear effect |
| increase masking ratio from 25% to 50% (ablation) | AST-T5 · HumanEval | Pass@1: 14.0 -> 14.3 | Helped |
| increase masking ratio from 25% to 50% (ablation) | AST-T5 · Bugs2Fix Small | EM: 23.8 -> 21.9 | Hurt |
| increase masking ratio from 25% to 50% (ablation) | AST-T5 · BigCloneBench | F1: 98.6 -> 97.1 | Hurt |
| increase masking ratio from 25% to 50% (ablation) | AST-T5 · All evaluation tasks (Avg) | average performance: 47.9 -> 46.4 | Hurt |
Surprising findings
- Raising the mask ratio to 50% gave only a marginal HumanEval gain (14.3 vs 14.0) while hurting transpilation and understanding, so 25% was chosen.
Limitations
- Masking is restricted to code AST subtrees, so natural-language generation (e.g., docstrings, summarization) may suffer.
- Assumes input code is syntactically parsable, which may limit applicability to incomplete or incorrect code.
- Pretraining at 277M parameters only; scalability to larger models and datasets is untested.
Experimental setup
Datasets: The Stack Dedup CodeSearchNet CodeXGLUE HumanEval MBPP Concode Bugs2Fix BigCloneBench Defect Detection
Benchmarks: HumanEval MBPP Concode Bugs2Fix Java-C# transpilation BigCloneBench Defect Detection HumanEval+ MBPP+ MBXP
Metrics: Pass@1 exact match F1 accuracy CodeBLEU
Baselines: T5 (own code-trained baseline) CodeT5 CodeT5+ StructCoder PLBART CodeBERT GraphCodeBERT GPT-J
Builds on: T5 CodeT5 Vanilla T5 span corruption Tree-sitter
Closest papers by meaning
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.87)
- Can Large Language Models Understand Intermediate Representations in Compilers? (ICML 2025 · similarity 0.87)
- CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules (ICLR 2024 · similarity 0.87)
- Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.86)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.86)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.