Research map / Architectures and efficiency / CNN design
PLEIADES: Building Temporal Kernels with Orthogonal Polynomials
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PLEIADES, causal (1+2)D spatiotemporal networks whose temporal kernels are linear combinations of Jacobi polynomial bases, enabling resampling of bin size without finetuning and achieving SOTA on DVS128 Gesture, AIS2024 eye tracking, and GEN4 detection with few parameters.
Problem
Efficient, low-latency online spatiotemporal classification and detection on event-based data with long temporal kernels, where standard explicit temporal convolutions are memory-heavy and unstable and cannot easily adapt to different sampling rates.
Approach
Parameterize each temporal convolution kernel as a trainable weighted sum of Jacobi polynomial basis functions (alpha=beta=-0.25, degree up to 4), discretized by integrating over time bins; this allows resampling the kernel to different bin sizes without retraining and enables contraction-order optimization that reduces compute and memory.
Key results
- On DVS128 Gesture, PLEIADES reaches 99.59% accuracy with 192K parameters (100% with output majority filtering), above Conv(1+2)D at 99.17% with 196K parameters.
- On AIS2024 3ET+, PLEIADES + CenterNet achieves 99.58% p10 with 277K parameters, versus 99.00% for Conv(1+2)D with 1.1M parameters.
- On Prophesee GEN4, PLEIADES + CenterNet obtains 0.556 mAP with 0.576M parameters, compared with 0.484 for GET-T at 21.9M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace free temporal kernels with orthogonal-polynomial kernels | Conv(1+2)D on DVS128 · DVS128 test | accuracy: 99.17 -> 99.59 | Helped |
| Add 0.15-second majority output filtering | PLEIADES on DVS128 · DVS128 test | accuracy: 99.59 (0.02) -> 100.00 (0.00) | Helped |
| Use PLEIADES instead of Conv(1+2)D | AIS2024 3ET+ · AIS2024 3ET+ | p10: 99.00 -> 99.58 (0.03) | Helped |
| Use PLEIADES instead of Conv(1+2)D | AIS2024 3ET+ · AIS2024 3ET+ | p5: 97.97 -> 97.95 (0.03) | No clear effect |
| Use PLEIADES instead of Conv(1+2)D | AIS2024 3ET+ · AIS2024 3ET+ | p3: 94.58 -> 94.94 (0.04) | Helped |
| Use PLEIADES + CenterNet instead of GET-T | Prophesee GEN4 · Prophesee GEN4 | mAP: 0.484 -> 0.556 | Helped |
| Use PLEIADES + CenterNet instead of SimCLR (ResNet50) | KITTI 2DOD · KITTI 2DOD | mAP: 0.570 -> 0.576 | Helped |
| Use polynomial degree 4 instead of degree 2 (ablation) | PLEIADES · DVS128 at 200 ms latency | accuracy: 55.4 (0.03) -> 73.2 (0.03) | Helped |
| Use polynomial degree 4 instead of degree 5 (ablation) | PLEIADES · DVS128 at 200 ms latency | accuracy: 67.6 (0.04) -> 73.2 (0.03) | Helped |
| Use polynomial degree 4 instead of degree 8 (ablation) | PLEIADES · DVS128 at 200 ms latency | accuracy: 60.2 (0.02) -> 73.2 (0.03) | Helped |
| Use polynomial degree 4 instead of degree 10 (ablation) | PLEIADES · DVS128 at 200 ms latency | accuracy: 52.5 (0.02) -> 73.2 (0.03) | Helped |
| Use polynomial degree 4 instead of free kernels (ablation) | PLEIADES · DVS128 at 200 ms latency | accuracy: 52.3 (0.02) -> 73.2 (0.03) | Helped |
Surprising findings
- Higher polynomial degree (10 or free kernels) degraded low-latency accuracy, so degree 4 performed best rather than more expressive kernels.
- Resampling trained kernels to different bin sizes (5 ms and 20 ms) without fine-tuning left accuracy-latency curves largely unchanged.
Limitations
- Memory cost of finite-window temporal buffers grows linearly with kernel length; recurrent online-basis reformulation is proposed but not implemented.
- Temporal expressivity is restricted relative to unstructured per-bin kernels, and polynomial degree is sensitive to tune.
- Warm-up latency of about 0.44 s on DVS128 from non-zero frame requirement; majority filtering adds latency.
Experimental setup
Datasets: DVS128 Gesture AIS2024 3ET+ eye tracking Prophesee GEN4 1Mpx automotive detection KITTI 2D object detection
Benchmarks: DVS128 Gesture AIS2024 3ET+ eye tracking (p10/p5/p3) Prophesee GEN4 (mAP for cars and pedestrians) KITTI 2DOD
Metrics: accuracy p10 p5 p3 mAP parameters MACs/sec latency
Baselines: Conv(1+2)D ANN-Rollouts TrueNorth CNN SLAYER MambaPupil CETM ERVT PEPNet RED Gray-RetinaNet S5-ViT-B GET-T
Builds on: Conv(1+2)D / R(2+1)D Legendre Memory Unit HiPPO CenterNet opt_einsum
Closest papers by meaning
- Rethinking Scale-Aware Temporal Encoding for Event-based Object Detection (NeurIPS 2025 · similarity 0.84)
- ALERT-Transformer: Bridging Asynchronous and Synchronous Machine Learning for Real-Time Event-based Spatio-Temporal Data (ICML 2024 · similarity 0.83)
- PASS: Path-selective State Space Model for Event-based Recognition (NeurIPS 2025 · similarity 0.83)
- FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies (NeurIPS 2025 · similarity 0.83)
- Recurrent Partial Kernel Network for Efficient Optical Flow Estimation (AAAI 2024 · similarity 0.82)
- UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio Video Point Cloud Time-Series and Image Recognition (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.