Research map / Time series / Foundation models for time series
Enhancing Masked Time-Series Modeling via Dropping Patches
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DropPatch, which randomly drops a fraction of sub-series patches before patch-level masked reconstruction in time-series pre-training, improving efficiency and forecasting across in-domain, cross-domain, few-shot and cold-start settings.
Problem
Masked time-series modeling with patch-level masking (PatchTST) suffers from superficial learning and over-fitting when mask ratio is low, and scattered attention and representation collapse when mask ratio is high.
Approach
Before masking, randomly drop a predefined proportion of patches (keeping positional encodings) so remaining patches are masked and reconstructed; dropped patches are excluded entirely from training for that epoch, unlike masked patches. This acts as data augmentation, cuts compute, and is argued theoretically to slow rank collapse of Transformer representations.
Key results
- In-domain forecasting: DropPatch average MSE/MAE on ETTm1 0.336/0.378 vs PatchTST 0.341/0.379; surpasses baselines in 13 of 14 metrics across 7 datasets (as claimed).
- Pre-training efficiency: on ETTm1, DropPatch uses 1404 MB and 32.2 s/epoch vs PatchTST 1722 MB and 44.5 s/epoch; on ECL 4256 MB / 306.7 s vs PatchTST 11050 MB / 528.5 s.
- Cold start with lookback 96 on Exchange: DropPatch 0.348/0.396 vs PatchTST 0.354/0.400.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DropPatch to PatchTST | PatchTST · In-domain forecasting across seven datasets | MSE: ETTm1: 0.341 -> 0.336; ETTm2: 0.258 -> 0.254; ETTh1: 0.430 -> 0.400; ETTh2: 0.355 -> 0.347; Weather: 0.225 -> 0.220; Tra | Helped |
| Apply DropPatch to PatchTST | PatchTST · ECL→ETTm1 and ECL→ETTm2 | MSE: ETTm1: 0.346 -> 0.349; ETTm2: 0.257 -> 0.258 | Hurt |
| Apply DropPatch to PatchTST | PatchTST · ECL→ETTh1, ETTh2, Weather, Traffic | MSE: ETTh1: 0.434 -> 0.395; ETTh2: 0.354 -> 0.350; Weather: 0.226 -> 0.222; Traffic: 0.411 -> 0.379 | Helped |
| Apply DropPatch to PatchTST | PatchTST · Cold start: Exchange, PEMS03, PEMS04, PEMS07, PEMS08 | MSE: Exchange: 0.354 -> 0.348; PEMS03: 0.205 -> 0.198; PEMS04: 0.273 -> 0.264; PEMS07: 0.219 -> 0.214; PEMS08: 0.233 -> 0.225 | Helped |
| Apply DropPatch to PatchTST | PatchTST pre-training · ETTm1 | memory (MB): 1722 -> 1404 | Helped |
| Apply DropPatch to PatchTST | PatchTST pre-training · ETTm1 | time per epoch (seconds): 44.5 -> 32.2 | Helped |
| Apply DropPatch to PatchTST | PatchTST · Few-shot forecasting on STS162M | MSE | Mixed |
| Apply DropPatch to PatchTST | PatchTST · STS66M cross-domain fine-tuning | MSE: Weather AVG: 0.224 -> 0.220; ETTh1 AVG: 0.411 -> 0.407; ETTh2 AVG: 0.358 -> 0.346; ETTm1 AVG: 0.351 -> 0.348; ETTm2 AVG: | Helped |
Surprising findings
- Dropping patches before masking gives a larger gain than simply increasing the mask ratio to match the number of visible patches (PatchTST(0.78) comparison in analysis).
- DropPatch shows reduced CKA similarity in cross-domain settings, which the authors interpret as better adaptation to domain shift.
Limitations
- Theoretical corollary relies on strong assumptions (small perturbations, uniform random row dropping) and the proof is heuristic with approximate steps.
- Cross-domain and synthesized-dataset results show PatchTST matching or beating DropPatch in several cells, and baselines were often run with differing configurations.
Experimental setup
Datasets: ETTh1 ETTh2 ETTm1 ETTm2 Weather ECL Traffic Exchange PEMS03 PEMS04 PEMS07 PEMS08
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Weather ECL Traffic Exchange PEMS03 PEMS04 PEMS07 PEMS08
Metrics: MSE MAE memory usage time per epoch
Baselines: PatchTST SimMTM Ti-MAE TST LaST CoST TS2Vec iTransformer DLinear FEDformer TF-C
Builds on: PatchTST masked autoencoder SimMTM
Closest papers by meaning
- Learning to Embed Time Series Patches Independently (ICLR 2024 · similarity 0.86)
- Content-aware Balanced Spectrum Encoding in Masked Modeling for Time Series Classification (AAAI 2025 · similarity 0.84)
- Discretizing Continuous Time Series for Imputation with Masked Diffusion Training (NeurIPS 2026 · similarity 0.84)
- Language Pre-training Guided Masking Representation Learning for Time Series Classification (AAAI 2025 · similarity 0.83)
- A Diffusion Model for Regular Time Series Generation from Irregular Data with Completion and Masking (NeurIPS 2025 · similarity 0.83)
- Abstain Mask Retain Core: Time Series Prediction by Adaptive Masking Loss with Representation Consistency (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.