Research map / Architectures and efficiency / Attention and transformers
Learning Language Representations with Logical Inductive Bias
Jianshu Chen
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FOLNet Base instead of BERT Base | language representation pretraining · GLUE dev, 16GB pretraining | Avg: 83.1 -> 87.7 | Improved |
| use FOLNet Base instead of DeBERTa Base | language representation pretraining · SQuAD 2.0 dev, 16GB pretraining | EM: 79.3 -> 84.7 | Improved |
| use FOLNet Base instead of DeBERTa Base | language representation pretraining · SQuAD 2.0 dev, 16GB pretraining | F1: 82.5 -> 87.9 | Improved |
| use FOLNet Base instead of RoBERTa Base | language representation pretraining · GLUE dev, 160GB pretraining | Avg: 86.4 -> 89.2 | Improved |
| use FOLNet Base instead of RoBERTa Base | language representation pretraining · FOLIO validation, 160GB pretraining | accuracy: 64.7 -> 64.2 | Hurt |
| use FOLNet Large instead of RoBERTa Large | language representation pretraining · FOLIO validation, 160GB pretraining | accuracy: 67.7 -> 70.6 | Improved |
| use the dual-branch j.a architecture instead of BERT (ablation) | FOLNet · GLUE dev | Avg: 83.1 -> 84.0 | Improved |
| use proposed relative positional encoding instead of absolute positional encoding (ablation) | FOLNet j.a · GLUE dev | Avg: 84.0 -> 84.7 | Improved |
| replace proposed relative positional encoding with T5 relative positional encoding (ablation) | FOLNet j.a · GLUE dev | Avg: 84.7 -> 84.7 | No clear effect |
| add mu and prod operators (ablation) | FOLNet j.a · GLUE dev | Avg: 84.7 -> 85.4 | Improved |
| increase binary predicate dimension D2 from 12 to 64 (ablation) | FOLNet j.a · GLUE dev | Avg: 84.0 -> 85.1 | Improved |
| add trans, prod, mu, and c operators to j.a (ablation) | FOLNet · GLUE dev | Avg: 85.1 -> 87.7 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- DeBERTa Large — DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION (ICLR 2021) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024 · similarity 0.85)
- Training Neural Networks as Recognizers of Formal Languages (ICLR 2025 · similarity 0.85)
- Inductive Learning of Logical Theories with LLMs: A Expressivity-graded Analysis (AAAI 2025 · similarity 0.85)
- Learning to Disentangle Latent Reasoning Rules with Language VAEs: A Systematic Study (AAAI 2026 · similarity 0.85)
- Logically Consistent Language Models via Neuro-Symbolic Integration (ICLR 2025 · similarity 0.84)
- How do Transformers Learn Implicit Reasoning? (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
