Research map / Time series / Foundation models for time series
Content-aware Balanced Spectrum Encoding in Masked Modeling for Time Series Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a content-aware balanced decoder (CBD) with frequency-domain interaction modulation and Bernstein-based spectrum energy rebalancing, plus a dual-constraint loss, to improve transformer masked time-series modeling for classification.
Problem
Transformer-based masked time-series modeling suffers from feature homogenization (rank collapse from long-dependency averaging) and spectrum energy imbalance (bias toward low-frequency components), hurting time-series classification representations.
Approach
An auxiliary frequency-domain decoder with two units: Content-aware Interaction Modulation (a dynamic, content-dependent spectral filter acting as convolution) to limit interaction scope and fight rank collapse, and Spectrum Energy Rebalance (learned Bernstein polynomial mapping of normalized spectral energy) to balance frequency energy; trained jointly with temporal decoder via a dual-constraint
Key results
- Linear probing accuracy on HAR reaches 93.78% and the method is best on most of the ten datasets, surpassing the second-best by 2.38% on HAR.
- Adding CBD to CRT raises average linear-probing accuracy from 67.64% to 72.95% (+5.31) across ten datasets.
- Ablation on HAR: using both temporal and frequency reconstruction losses gives 93.37% LP versus 89.47% for temporal-only loss; full model reaches 93.78%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the CBD two-pronged spectrum decoder to the temporal masked-modeling framework | Ours · HAR linear evaluation | accuracy: 91.40$\pm$0.16 -> 93.78$\pm$0.70 | Helped |
| Add the CBD two-pronged spectrum decoder to the temporal masked-modeling framework | Ours · HAR fine-tune evaluation | accuracy: 95.11$\pm$0.18 -> 95.92$\pm$0.78 | Helped |
| Add the dual-constraint frequency loss to temporal reconstruction (ablation) | Ours · HAR linear evaluation | accuracy: 89.47 -> 91.68 | Helped |
| Use frequency reconstruction instead of temporal reconstruction (ablation) | Ours · HAR linear evaluation | accuracy: 89.47 -> 91.61 | Helped |
| Add both temporal and frequency decoders with dual-constraint losses (ablation) | Ours · HAR linear evaluation | accuracy: 93.37 -> 93.78 | Helped |
| Add CIM (ablation) | Ours CBD · HAR linear evaluation | accuracy: 93.45 -> 93.78 | Helped |
| Add SER (ablation) | Ours CBD · HAR linear evaluation | accuracy: 91.92 -> 93.78 | Helped |
| Set Bernstein polynomial order K to 12 rather than K=1 (ablation) | SER · HAR linear evaluation | accuracy: 92.39 -> 93.78 | Helped |
| Increase Bernstein polynomial order from K=12 to K=16 (ablation) | SER · HAR linear evaluation | accuracy: 93.78 -> 93.48 | Hurt |
| Add CBD to PatchTST | PatchTST · 10-dataset average linear evaluation | accuracy: +1.50 | Helped |
| Add CBD to CRT | CRT · 10-dataset average linear evaluation | accuracy: +5.31 | Helped |
| Add CBD to SimMTM | SimMTM · 10-dataset average fine-tune evaluation | accuracy: +0.59 | Helped |
Surprising findings
- Adding CBD to TimeMAE degraded performance, which the authors attribute to TimeMAE not reconstructing from original data, giving the CBD inaccurate supervision.
- The frequency-domain decoder alone with a dual-constraint loss improved linear accuracy on HAR, and the learned interaction scope trend differed across datasets (shrinking on HAR, expanding on PS and FM).
Limitations
- Performance gains are modest on several datasets, and some baselines (e.g., SimMTM on SRSCP1, CSL/TS-GAC) match or beat the method on individual tasks.
- TS-GAC results are missing on three datasets (ECG5000, FB, UWare) and TimesURL is excluded from fine-tuning evaluation.
- The dual-constraint weight gamma must be tuned per dataset, and the mask ratio and K value are chosen by hyperparameter search.
Experimental setup
Datasets: HAR PhonemeSpectra SelfRegulationSCP1 MotorImagery FingerMovements ArticularyWordRecognition SpokenArabicDigits ECG5000 FordB UWaveGestureLibraryAll
Benchmarks: HAR PhonemeSpectra SelfRegulationSCP1 MotorImagery FingerMovements ArticularyWordRecognition SpokenArabicDigits ECG5000 FordB UWaveGestureLibraryAll
Metrics: accuracy F1
Baselines: TS-TCC TS2Vec TS-GAC TimesURL CSL PatchTST CRT SimMTM TimeMAE
Builds on: TimeMAE masked autoencoder SimMTM CRT
Closest papers by meaning
- Language Pre-training Guided Masking Representation Learning for Time Series Classification (AAAI 2025 · similarity 0.86)
- Frequency-Masked Embedding Inference: A Non-Contrastive Approach for Time Series Representation Learning (AAAI 2025 · similarity 0.86)
- TSLANet: Rethinking Transformers for Time Series Representation Learning (ICML 2024 · similarity 0.86)
- Learning to Embed Time Series Patches Independently (ICLR 2024 · similarity 0.86)
- Abstain Mask Retain Core: Time Series Prediction by Adaptive Masking Loss with Representation Consistency (NeurIPS 2025 · similarity 0.85)
- OTIS: Learning High-Quality Time Series Features With Tiny Encoders (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.