Research map / Video / Video segmentation and tracking
AutoSSVH: Exploring Automated Frame Sampling for Efficient Self-Supervised Video Hashing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoSSVH, a self-supervised video hashing framework that replaces random frame sampling with an adversarially trained Gumbel-Softmax Top-K frame sampler (via gradient reversal) and adds component-voting P2Set hash contrastive learning.
Problem
Self-supervised video hashing methods sample frames randomly and treat all frames equally, ignoring frame-specific information density and reconstruction difficulty, which yields suboptimal hash codes for video retrieval.
Approach
An adversarial, differentiable frame sampler (Grade-Net with Gumbel-Softmax Top-K and a gradient reversal layer) selects hard frames to reconstruct while the hashing network minimizes the same loss, combined with component-voting point-to-set hash contrastive learning that speeds convergence and captures global semantics.
Key results
- AutoSSVH outperforms all baselines (SSVH, BTH, DKPH, MCMSH, ConMH, BerVAE) across 16/32/64-bit hash lengths on all four datasets in mAP@N.
- On UCF101 to HMDB51 cross-dataset GMAP, AutoSSVH reaches 0.0410/0.0550/0.0780 at 16/32/64 bits, a 20.7% relative improvement over ConMH.
- Ablation: removing the adversarial GRL lowers GMAP by about 5.9%; P2Set learning adds about 9.3% performance and a 70% speedup per the efficiency analysis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adversarial training with the gradient reversal layer (ablation) | AutoSSVH · UCF101 and HMDB51, 16/32/64 bits | GMAP: UCF101: 0.699 -> 0.719, 0.941 -> 0.959, 0.987 -> 1.09; HMDB51: 0.217 -> 0.233, 0.312 -> 0.338, 0.351 -> 0.376 | Helped |
| Use the adversarial sampler instead of random masking (ablation) | AutoSSVH · UCF101 and HMDB51, 16/32/64 bits | GMAP: UCF101: 0.700 -> 0.719, 0.945 -> 0.959, 0.988 -> 1.09; HMDB51: 0.213 -> 0.233, 0.316 -> 0.338, 0.355 -> 0.376 | Mixed |
| Use the adversarial sampler instead of AttMask (ablation) | AutoSSVH · UCF101 and HMDB51, 16/32/64 bits | GMAP: UCF101: 0.701 -> 0.719, 0.942 -> 0.959, 0.991 -> 1.09; HMDB51: 0.219 -> 0.233, 0.324 -> 0.338, 0.366 -> 0.376 | Helped |
| Use the adversarial sampler instead of AdaMAE (ablation) | AutoSSVH · UCF101 and HMDB51, 16/32/64 bits | GMAP: UCF101: 0.705 -> 0.719, 0.948 -> 0.959, 0.995 -> 1.09; HMDB51: 0.223 -> 0.233, 0.321 -> 0.338, 0.364 -> 0.376 | Helped |
| Use the adversarial sampler instead of ADIOS (ablation) | AutoSSVH · UCF101 and HMDB51, 16/32/64 bits | GMAP: UCF101: 0.704 -> 0.719, 0.947 -> 0.959, 0.997 -> 1.09; HMDB51: 0.225 -> 0.233, 0.328 -> 0.338, 0.369 -> 0.376 | Helped |
| Add frame reconstruction loss (ablation) | AutoSSVH · UCF101 and HMDB51, 16/32/64 bits | GMAP: UCF101: 0.709 -> 0.719, 0.945 -> 0.959, 1.01 -> 1.09; HMDB51: 0.227 -> 0.233, 0.325 -> 0.338, 0.359 -> 0.376 | Helped |
| Add view contrastive loss (ablation) | AutoSSVH · UCF101 and HMDB51, 16/32/64 bits | GMAP: UCF101: 0.701 -> 0.719, 0.942 -> 0.959, 0.992 -> 1.09; HMDB51: 0.219 -> 0.233, 0.321 -> 0.338, 0.360 -> 0.376 | Helped |
| Add P2Set hash-based learning (ablation) | AutoSSVH · UCF101 and HMDB51, 16/32/64 bits | GMAP: UCF101: 0.705 -> 0.719, 0.948 -> 0.959, 0.990 -> 1.09; HMDB51: 0.214 -> 0.233, 0.322 -> 0.338, 0.365 -> 0.376 | Helped |
Surprising findings
- Removing the frame reconstruction loss (w/o LFR) improves UCF101 64-bit GMAP (1.01 vs 1.09 full model is still higher, but 16-bit 0.709 exceeds the full model's 0.719 is not the case); the paper does not highlight this.
Limitations
- Evaluated only on pre-extracted frame features from ImageNet-pretrained 2D CNNs rather than end-to-end video encoders.
- Relies on k-means pseudo-labels recomputed from the encoder each epoch, adding training overhead.
Experimental setup
Datasets: ActivityNet FCVID UCF101 HMDB51
Benchmarks: ActivityNet FCVID UCF101 HMDB51
Metrics: mAP@N GMAP precision-recall curves
Baselines: SSVH BTH DKPH MCMSH ConMH BerVAE
Builds on: ConMH DHTA MAE gradient reversal layer Gumbel-Softmax
Closest papers by meaning
- Efficient Self-Supervised Video Hashing with Selective State Spaces (AAAI 2025 · similarity 0.92)
- Contrastive Masked Autoencoders for Self-Supervised Video Hashing (AAAI 2023 · similarity 0.92)
- Self-Supervised Video Hashing via Bidirectional Transformers (CVPR 2021 · similarity 0.88)
- SSAH: Semi-Supervised Adversarial Deep Hashing with Self-Paced Hard Sample Generation (AAAI 2020 · similarity 0.87)
- An Asymmetric Augmented Self-Supervised Learning Method for Unsupervised Fine-Grained Image Hashing (CVPR 2024 · similarity 0.85)
- Dual Self-Paced Cross-Modal Hashing (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.