Research map / Large language models / Language, safety and interpretability
Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FANformer, which injects Fourier Analysis Network periodicity modeling into the attention QKV projections, and shows gains in scaling, pretraining (1B on 1T tokens), and rule-based reasoning over Transformers.
Problem
Transformers have weak periodicity modeling, which hurts learning efficiency and the learning of underlying rules for LLMs.
Approach
Adapts the Fourier Analysis Network (FAN) layer into the attention feature projection (ATF module) so that Q, K, V are computed from a representation containing cos/sin periodic features, with a hyperparameter p setting the periodic proportion; equivalent to standard attention applied to a FAN-transformed input.
Key results
- FANformer matches Transformer performance with 69.2% of parameters or 79.7% of training tokens (159.6B vs 200B tokens).
- FANformer-1B averages 65.6 on 8 commonsense tasks vs OLMo-1B 60.4 (about 8.8% relative gain with half the tokens).
- On ULogic stress test, FANformer-1B reaches 38.2% accuracy vs OLMo-1B 0.0% and Qwen2.5-1.5B 7.1%, with GPT-4 at 65.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard Transformer attention with FANformer ATF | 1B language models · V2 and V3 validation sets | PPL: 30.20 -> 29.40; 24.28 -> 23.62 | Helped |
| Replace standard Transformer attention with FANformer ATF | 1B language models · Downstream benchmarks | average accuracy: 53.10 -> 55.19 | Helped |
| Use FANformer architecture rather than OLMo-1B | 1.1B pretrained LLM using 1T tokens · GSM8K | score: 8.9 -> 15.7 | Helped |
| Use FANformer-1B rather than OLMo-1B | ULogic reasoning · ULogic (two most challenging levels) | accuracy: 0.0 -> 38.2 | Helped |
| Use FANformer-1B-SFT rather than OLMo-1B-SFT | supervised fine-tuned LLMs · ToxiGen | % Toxic: 2.8 -> 10.4 | Hurt |
| Replace the FAN attention layer with an MLP (ablation) | FANformer · V2 Eval | PPL: 29.40 -> 30.31 | Hurt |
| Use two linear transforms for QKV projection instead of FANformer ATF (ablation) | FANformer · V2 Eval | PPL: 29.40 -> 29.68 | Hurt |
| Use the original FAN layer with activation in attention (ablation) | FANformer · V2 Eval | PPL: 29.40 -> 30.64 | Hurt |
| Move FAN periodicity modeling from attention into the FFN (ablation) | FANformer · Downstream benchmarks | average accuracy: 55.19 -> 53.95 | Hurt |
| Add activation functions to FANformer attention (ablation) | FANformer · Downstream benchmarks | average accuracy: 55.19 -> 53.61 | Hurt |
Surprising findings
- Transformer+ATM (an MLP in place of the FAN layer) gives no gain over the standard Transformer, indicating the gains come from the FAN design rather than extra depth.
- R1-Distill-Qwen1.5B performs much worse than its base on general commonsense tasks, which the authors attribute to distillation shortcomings.
Limitations
- Pretraining compute of about 47,600 GPU hours is costly to replicate.
- FANformer-1B's ULogic accuracy (38.2%) remains well below GPT-4 (65.1%).
- Optimal p varies with model size, requiring tuning of the hyperparameter.
Experimental setup
Datasets: Dolma Dolma v1_6-sample tulu-3-sft-olmo-2-mixture ULogic modular addition and linear regression synthetic tasks
Benchmarks: ARC-C ARC-E BoolQ HellaSwag OBQA PIQA SciQ WinoGrande MMLU AlpacaEval ToxiGen TruthfulQA
Metrics: perplexity loss accuracy average accuracy win rate % toxic Lipschitz constant
Baselines: Transformer Transformer+ATM Transformer+ATL OLMo-1B Qwen2.5-1.5B Llama-3.2-1B TinyLlama-v1.1 OpenLLaMA-v2-3B StableLM-v2-3B R1-Distill-Qwen1.5B GPT-4
Builds on: Fourier Analysis Network OLMo FlashAttention SwiGLU
Closest papers by meaning
- Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model (CVPR 2025 · similarity 0.88)
- Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance (NeurIPS 2024 · similarity 0.86)
- Who Reasons in the Large Language Models? (NeurIPS 2025 · similarity 0.85)
- Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.85)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.85)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.