Research map / Architectures and efficiency / Pruning and efficient tuning
Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffEqFormer, a transformer formulated as a non-autonomous neural ODE whose attention and feed-forward weights are generated from a continuous layer index, and analyzes its spectral dynamics, Lyapunov-based token sensitivity, and adaptive fine-tuning across depths.
Problem
Existing neural-ODE transformer theories assume shared weights across depth, which predicts clustering of outputs at infinite depth, conflicting with autoregressive next-token prediction; also no pretrained model supports fine-tuning across different depths/architectures.
Approach
Generate all attention and feed-forward weights as functions of continuous time via a sinusoidal time embedding plus MLP and projection, yielding a non-autonomous ODE transformer; analyze eigenvalue growth of QK and OV pairs, use Lyapunov exponents for token-level sensitivity, and fine-tune by changing ODE step size (number of layers) with LoRA or full-rank updates.
Key results
- On OpenWebText, DiffEqFormer-large reaches 15.43 perplexity vs 15.64 for GPT-large; its Llama-1B variant reaches 9.68 vs 10.17.
- On WikiText103 medium-size, DiffEqFormer obtains 21.94 perplexity vs 23.38 for GPT-medium.
- Under LoRA fine-tuning, models pretrained with 18 steps and fine-tuned with 12, 18, or 24 steps outperform the 24-layer GPT baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace GPT-small with DiffEqFormer-small | autoregressive language modeling · WikiText103 | perplexity: 22.87 -> 22.84 | Helped |
| Replace GPT-medium with DiffEqFormer-medium | autoregressive language modeling · WikiText103 | perplexity: 23.38 -> 21.94 | Helped |
| Replace GPT with DiffEqFormer at matched sizes | autoregressive language modeling · OpenWebText | perplexity: 22.60 -> 22.06; 17.85 -> 17.67; 15.64 -> 15.43 | Helped |
| Replace Llama-1B with DiffEqFormer-Llama-1B | autoregressive language modeling · OpenWebText | perplexity: 10.17 -> 9.68 | Helped |
| Use DiffEqFormer instead of GPT-large | GPT-large configuration · Lambada OpenAI zero-shot | accuracy: 37.24 -> 38.39 | Helped |
| Use DiffEqFormer instead of GPT-large | GPT-large configuration · Lambada Standard zero-shot | accuracy: 29.30 -> 30.37 | Helped |
| Use DiffEqFormer instead of GPT-large | GPT-large configuration · Lambada Standard five-shot | accuracy: 23.05 -> 24.82 | Helped |
| Use DiffEqFormer instead of Llama-1B | Llama-1B configuration · Lambada OpenAI zero-shot | accuracy: 41.84 -> 43.74 | Helped |
| Use DiffEqFormer instead of Llama-1B | Llama-1B configuration · Lambada OpenAI five-shot | accuracy: 30.47 -> 34.41 | Helped |
| Use DiffEqFormer instead of Llama-1B | Llama-1B configuration · Lambada Standard zero-shot | accuracy: 27.58 -> 31.55 | Helped |
| Use DiffEqFormer instead of Llama-1B | Llama-1B configuration · Lambada Standard five-shot | accuracy: 24.04 -> 28.22 | Helped |
| Set Adam beta2 to 0.95 instead of 0.999 (ablation) | DiffEqFormer · OpenWebText validation, GPT-large configuration | perplexity: 15.89 -> 15.43 | Helped |
Surprising findings
- Eigenvalue magnitudes of QK and OV pairs increase with depth, peaking near the last layers, contrary to the shared-weight clustering assumption; simulations with time-dependent weights show no clusters.
- Setting Adam beta2 to 0.95 instead of 0.999 improved DiffEqFormer perplexity while GPT performed better with 0.999.
Limitations
- Parameter count is much higher than GPT counterparts (e.g., 1.86B for d_emb=144 medium config) due to time-dependent weight generator.
- Shared-weight-based clustering theory gap is shown only via simulation of attention-only models, not formally proven.
- Downstream gains are small and mostly limited to Lambada tasks; most tasks comparable.
Experimental setup
Datasets: OpenWebText WikiText103 lm-evaluation-harness tasks
Benchmarks: WikiText103 OpenWebText Lambada OpenAI Lambada Standard ARC-Challenge ARC-Easy HellaSwag LogiQA MMLU PIQA SciQ Winogrande
Metrics: perplexity accuracy
Baselines: GPT-small GPT-medium GPT-large Llama-1B shared-parameter transformer
Builds on: neural ODE MacaronNet GPT-2 LoRA Levanter
Closest papers by meaning
- Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning (NeurIPS 2024 · similarity 0.88)
- Understanding Differential Transformer Unchains Pretrained Self-Attentions (NeurIPS 2025 · similarity 0.87)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.87)
- Global Convergence in Neural ODEs: Impact of Activation Functions (ICLR 2025 · similarity 0.86)
- ODEFormer: Symbolic Regression of Dynamical Systems with Transformers (ICLR 2024 · similarity 0.86)
- Wavy Transformer (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.