Research map / Continual learning / Class-incremental
Hypercorrelation Evolution for Video Class-Incremental Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hypercorrelation Evolution (HCE) for video class-incremental learning, combining a hierarchical aggregation strategy (4D-convolution over layer-wise cosine similarity matrices for multi-level distillation) with a correlation refinement mechanism that reweights cross-entropy by class-wise hypercorrelation variance ratios.
Problem
Video class-incremental learning must learn new action classes while avoiding catastrophic forgetting of old ones under limited exemplar memory; single-layer distillation and class-number-ratio rebalancing are poorly suited to video redundancy and multi-level sub-action semantics.
Approach
Aggregate layer-wise cosine similarity matrices between old and new backbone features with a sparse 4D convolution to form hypercorrelations used for multi-level distillation, and rebalance old/new classes in the cross-entropy loss using the log-variance of these hypercorrelations instead of sample-count ratios.
Key results
- On HMDB51 (5x5 stages) HCE reaches 48.63% CNN and 52.01% NME, versus 47.54% / 51.12% for FrameMaker.
- On Something-Something V2 (10x9 stages) HCE reaches 43.99% CNN accuracy versus 42.65% for FrameMaker; in the 5x18 setting it reaches 32.51% CNN vs 30.98%.
- On UCF101 (10x5 stages) HCE reaches 79.12% CNN and 80.01% NME, versus 78.13% / 78.64% for FrameMaker.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HCE instead of FrameMaker | Video class-incremental action recognition · UCF101, 10 × 5 stages | accuracy (CNN): 78.13 -> 79.12 | Helped |
| Add hierarchical distillation (Lk_hc) to Lk_ce + Lk_kd (ablation) | HCE · HMDB51, 5 × 5 stages | accuracy (CNN): 44.36 -> 47.34 | Helped |
| Add last-layer distillation (Lk_kd) to Lk_ce + Lk_hc (ablation) | HCE · HMDB51, 5 × 5 stages | accuracy (CNN): 45.45 -> 47.34 | Helped |
| Use CRM-weighted cross-entropy (L'k_ce) instead of unweighted cross-entropy (ablation) | HCE · HMDB51, 5 × 5 stages | accuracy (CNN): 47.34 -> 48.63 | Helped |
| Replace single-level distillation with HAS (ablation) | HCE · HMDB51, 5 steps | accuracy (CNN): 44.86 -> 48.63 | Helped |
| Replace multi-level TCD distillation with HAS (ablation) | HCE · HMDB51, 5 steps | accuracy (CNN): 45.34 -> 48.63 | Helped |
| Use CRM instead of old/new sample-number ratio (ablation) | HCE · HMDB51, 5 steps | accuracy (CNN): 46.88 -> 48.63 | Helped |
| Vary the expanding scalar from 100 to 1000 (ablation) | HCE · HMDB51 | accuracy (CNN): 47.88 -> 48.63 | Mixed |
| Vary the expanding scalar from 1000 to 2000 (ablation) | HCE · HMDB51 | accuracy (CNN): 48.63 -> 48.43 | No clear effect |
| Vary the expanding scalar from 1000 to 10000 (ablation) | HCE · HMDB51 | accuracy (CNN): 48.63 -> 48.47 | No clear effect |
Surprising findings
- Applying the number-ratio rebalancing from image CIL gave little effect in video CIL, attributed to frame redundancy.
- Single-layer (last-layer) distillation is helpful for classification but not for preserving video knowledge, per the paper's analysis.
Limitations
- Improvements over the prior SOTA (FrameMaker) are modest, around 2-3 points.
- Expansion scalar and decoder design are hyperparameter-dependent, though the paper reports robustness across a range of scalars.
- Evaluation limited to three action recognition benchmarks and ResNet-based backbones.
Experimental setup
Datasets: UCF101 HMDB51 Something-Something V2
Benchmarks: UCF101 HMDB51 Something-Something V2
Metrics: top-1 accuracy (CNN classifier) top-1 accuracy (NME classifier)
Baselines: Fine-tuning LwFMC LwM iCaRL UCIR PODNet TCD FrameMaker
Builds on: TCD FrameMaker PODNet iCaRL UCIR
Closest papers by meaning
- StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learning (ICLR 2026 · similarity 0.88)
- Hyperbolic Displacement-Constrained Adaptation for CLIP-Based Class-Incremental Learning (NeurIPS 2026 · similarity 0.86)
- Learning Conditional Space-Time Prompt Distributions for Video Class-Incremental Learning (CVPR 2025 · similarity 0.85)
- Task-recency bias strikes back: Adapting covariances in Exemplar-Free Class Incremental Learning (NeurIPS 2024 · similarity 0.84)
- Revisiting Generative Replay for Class Incremental Object Detection (CVPR 2025 · similarity 0.84)
- Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.