Research map / Probabilistic and scientific modeling / Differential equations and PDEs
Efficient Neural Controlled Differential Equations via Attentive Kernel Smoothing
Egor Serov, Ilya Kuleshov, Alexey Zaytsev
This paper tried a common change: More attention heads
increase number of attention heads from 1 to 8 · UEA classification (3 datasets) · error rate: error rates reduced as M increases from 1 to 8; plateau around M = 4 for GP Improved
What 107 accepted papers report for this change
- 62 improved
- 13 no clear effect
- 13 mixed
- 19 hurt
- Universal Approximation with Softmax Attention ICML 2026 · increase the number of attention heads H · synthetic truncated linear model (d=10, n=50) · test MSEImproved
- TCAP: Tri-Component Attention Profiling for Unsupervised Backdoor Detection in MLLM Fine-Tuning ICML 2026 · raise number of heads from 10 to 14 · ScienceQA SIG detection · F1: 89.74 -> 77.09Hurt
- In-Context Algorithm Emulation in Fixed-Weight Transformers ICLR 2026 · increase number of attention heads from 1 to 12 · Attention-head emulation · MSE: 3.469 -> 0.686Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add attention-based multi-view heads with GP smoothing (MVC-GP) instead of cubic-spline path | Neural CDE (Cubic) · ETTm1 (24-step horizon, 30% drops) | MSE: 0.487 -> 0.428 | Improved |
| use MVC GP instead of cubic-spline Neural CDE | Neural CDE (Cubic) · ETTm1 (24-step horizon, 30% drops) | training time (s): 625 -> 248 | Improved |
| use MVC GP instead of GRU | GRU · ETTm1 (24-step horizon, 30% drops) | MSE: 0.476 -> 0.428 | Mixed |
| increase number of attention heads from 1 to 8 (ablation) | MV-CDE / MVC-CDE · UEA classification (3 datasets) | error rate: error rates reduced as M increases from 1 to 8; plateau around M = 4 for GP | Improved |
| replace spline interpolation with kernel/GP smoothing under input noise | Linear/Cubic Neural CDE · noise robustness test | NFE and error rate: speedup 4.3 to 14.5 times over standard CDE methods | Improved |
Closest papers by meaning
- On the Generalization and Approximation Capacities of Neural Controlled Differential Equations (ICLR 2024 · similarity 0.87)
- Log Neural Controlled Differential Equations: The Lie Brackets Make A Difference (ICML 2024 · similarity 0.87)
- Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Models (NeurIPS 2025 · similarity 0.86)
- Neural Stochastic Flows: Solver-Free Modelling and Inference for SDE Solutions (NeurIPS 2025 · similarity 0.86)
- How Deep Do We Need: Accelerating Training and Inference of Neural ODEs via Control Perspective (ICML 2024 · similarity 0.85)
- CALM-PDE: Continuous and Adaptive Convolutions for Latent Space Modeling of Time-dependent PDEs (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
