Research map / Architectures and efficiency / Attention and transformers
PredToken: Predicting Unknown Tokens and Beyond with Coarse-to-Fine Iterative Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PredToken, a VQGAN-based spatiotemporal predictive framework that uses wavelet-based decomposition into coarse/fine tokens and coarse-to-fine iterative decoding with dynamic and static soft decoding guided by a mask discriminator.
Problem
Predictive learning of future frames in unannotated spatiotemporal data requires preserving low-level detail and capturing high-level dynamics, but existing discrete-token approaches struggle with token representation quality and irreversible, independent token decoding.
Approach
Decompose frames via DWT into low- and high-frequency tokens, quantize with a dimension-aware (time/space/channel attention) quantizer, and decode coarse tokens with dynamic soft decoding (mask discriminator re-masks wrong tokens) followed by fine tokens with static soft decoding (entropy-based scores, soft embedding averaging).
Key results
- On TaxiBJ, PredToken reaches SSIM 0.985 and PSNR 39.79, versus 39.50 PSNR for the best baseline TAU.
- On KTH (10→20 frames), LPIPS drops from 0.2394 (TAU) to 0.1265 and PSNR rises to 35.11 over TAU's 34.13.
- On SJTU4K at t+3, PSNR improves from 21.03 (TAU) to 23.89 and LPIPS falls from 65.21 to 61.37.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PredToken instead of TAU | Spatiotemporal prediction models · TaxiBJ (4 → 4) | PSNR: 39.50 -> 39.79 | Helped |
| Use PredToken instead of TAU | Spatiotemporal prediction models · WeatherBench (12 → 12) | MAE: 0.6810 -> 0.6347 | Helped |
| Use PredToken instead of TAU | Spatiotemporal prediction models · KTH (10 → 20) | LPIPS: 0.2394 -> 0.1265 | Helped |
| Use PredToken instead of TAU | Spatiotemporal prediction models · SJTU4K (4 → 4), t+3 | PSNR: 21.03 → 23.89 | Helped |
| Add DQR to the VQGAN tokenizer (ablation) | PredToken tokenizer · DAVIS17 tokenizer reconstruction | SSIM: 80.4 -> 89.3 | Helped |
| Add dimension-aware quantization (DQM) to DQR (ablation) | PredToken tokenizer · DAVIS17 tokenizer reconstruction | SSIM: 89.3 -> 89.7 | Helped |
| Add shared codebook (SC) to DQR + DQM (ablation) | PredToken tokenizer · DAVIS17 tokenizer reconstruction | SSIM: 89.7 -> 89.8 | Helped |
| Add soft decoding (SD) (ablation) | PredToken iterative decoding · KTH | SSIM: 87.8 -> 88.5 | Helped |
| Add mask discriminator (MD) (ablation) | PredToken iterative decoding · KTH | SSIM: 87.8 -> 89.4 | Helped |
| Add both soft decoding and mask discriminator (ablation) | PredToken iterative decoding · KTH | PSNR: 31.97 -> 33.03 | Helped |
| Use dynamic soft decoding for coarse tokens and static soft decoding for fine tokens (ablation) | PredToken iterative decoding · KTH | PSNR | Helped |
Surprising findings
- Using DSD+DSD for both levels is theoretically optimal but its extra training objectives slow loss convergence, so DSD (coarse) + SSD (fine) was chosen.
Limitations
- Coarse level requires more decoding iterations than the fine level.
- Adversarial mask discriminator adds training complexity; DSD+DSD converges slowly.
Experimental setup
Datasets: TaxiBJ WeatherBench KTH Kitti&Caltech UCF101 SJTU4K DAVIS17
Benchmarks: TaxiBJ WeatherBench KTH Kitti&Caltech UCF101&DAVIS SJTU4K
Metrics: SSIM PSNR MAE RMSE LPIPS MSE
Baselines: ConvLSTM PredRNN PredRNN++ E3D-LSTM PredRNNv2 SimVP TAU SwinLSTM DMVFN MIM MAU
Builds on: VQGAN MaskGIT MaskViT MAGVIT Token-Critic
Closest papers by meaning
- TwinPrune: Density-Aware Two-Phase Token Pruning for Vision-Language Models (NeurIPS 2026 · similarity 0.87)
- VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models (NeurIPS 2025 · similarity 0.87)
- DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models (CVPR 2025 · similarity 0.86)
- Epsilon-VAE: Denoising as Visual Decoding (ICML 2025 · similarity 0.86)
- Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation (NeurIPS 2025 · similarity 0.86)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.