Research map / Video / Video segmentation and tracking
Exploring Domain Incremental Video Highlights Detection with the LiveFood Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces incremental video highlights detection and the LiveFood gourmet dataset, and proposes Global Prototype Encoding (GPE), a prototype-based classifier with a dual-constrained prototype drift limit for domain-incremental learning.
Problem
Video highlights detection assumes a closed world with fixed highlight categories and all training data available upfront, so models cannot scale to new highlight domains or data without retraining on everything.
Approach
GPE classifies each frame by L2 distance to two groups of learnable prototypes (highlight and vanilla) on top of a ConvNeXt and transformer encoder, and constrains prototype drift between incremental stages via a Lagrangian dual, avoiding replay buffers and growing architectures.
Key results
- On LiveFood, GPE (dynamic mode with constraint, GPE*) reaches 34.05 average mAP versus 30.52 for the lower bound and 36.68 for the upper bound; the fixed mode GPE(Mf) gains 2.91% over the lower bound.
- On R-MNIST (20 tasks, MLP), dynamic GPE achieves 85.42% average accuracy without a buffer and 90.17% with a 200-sample buffer, versus 90.04% for DER.
- With ResNet-18 and no buffer, GPE reaches 94.77% top-1 on R-MNIST; buffers of 200 and 500 samples raise it by 2.29 and 3.56 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPE in fixed mode instead of the lower-bound unconstrained training | GPE on LiveFood · LiveFood test, Avg. | mAP: 30.52 -> 33.43 | Helped |
| Use GPE (M_f) instead of SI | GPE on LiveFood · LiveFood test, Avg. | mAP: 31.86 -> 33.43 | Helped |
| Use GPE (M_f) instead of DER with the same replay scheme | GPE on LiveFood · LiveFood test, Avg. | mAP: 32.99 -> 33.43 | Helped |
| Use dynamic GPE with a 200-sample memory buffer instead of DER | GPE (M_d) on R-MNIST · R-MNIST, 20 tasks, linear classifier | Avg Acc.: 90.04 -> 90.17 | Helped |
| Increase the initial prototype count from k=10 to k=50 (ablation) | GPE (M_f) on LiveFood · LiveFood, Avg., γ=5.0 | mAP: 30.99 -> 33.48 | Helped |
| Increase the initial prototype count from k=40 to k=50 (ablation) | GPE (M_f) on LiveFood · LiveFood, Avg., γ=5.0 | mAP: 33.43 -> 33.48 | No clear effect |
| Increase prototype-change constraint γ from 1e-3 to 5.0 (ablation) | GPE (M_f) on LiveFood · LiveFood, Avg., k=40 | mAP: 31.75 -> 33.43 | Helped |
| Increase prototype-change constraint γ from 5.0 to 15.0 (ablation) | GPE (M_f) on LiveFood · LiveFood, Avg., k=40 | mAP: 33.43 -> 32.50 | Hurt |
| Add a 200-sample memory buffer | GPE (M_d) with ResNet-18 on R-MNIST · R-MNIST, 20 tasks, convolution-based models | top-1 classification accuracy: +2.29% | Helped |
| Add a 500-sample memory buffer | GPE (M_d) with ResNet-18 on R-MNIST · R-MNIST, 20 tasks, convolution-based models | top-1 classification accuracy: +3.56% | Helped |
Surprising findings
- Replay-based DER with a buffer underperforms the proposed replay-free GPE on LiveFood in some stages, and the paper attributes DER's forgetting to limited buffer size.
- Too large a distance constraint (gamma=15) causes a near 1% mAP drop due to forgetting, while too small a constraint prevents learning new domains.
Limitations
- Gains over the strongest replay baseline are limited on LiveFood; the upper bound remains well above GPE (36.68 vs 34.05).
- Dataset is restricted to gourmet videos, and licensing is non-commercial only.
- Hyperparameters such as k and gamma are tuned on the test-set-like evaluation and the constraint upper bound is set empirically.
Experimental setup
Datasets: LiveFood R-MNIST ImageNet
Benchmarks: LiveFood R-MNIST
Metrics: mAP average accuracy top-1 accuracy
Baselines: SI oEWC ER DER GEM FDR GSS HAL A-GEM DER++ Co2L
Builds on: ConvNeXt Moment-DETR Co2L DER
Closest papers by meaning
- Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection (CVPR 2024 · similarity 0.84)
- Aha! - Predicting What Matters Next: Online Highlight Detection Without Looking Ahead (NeurIPS 2025 · similarity 0.84)
- Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection (AAAI 2025 · similarity 0.84)
- HD-EPIC: A Highly-Detailed Egocentric Video Dataset (CVPR 2025 · similarity 0.84)
- ShotVL: Human-Centric Highlight Frame Retrieval via Language Queries (AAAI 2025 · similarity 0.83)
- OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.