Research map / Video / Action recognition
BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts image-pretrained CLIP to video action recognition by aligning all visual and textual tokens with Brownian distance correlation instead of cosine similarity between global tokens.
Problem
Existing CLIP-based video action recognition methods align video and text with cosine similarity over global tokens, which captures only linear, coarse relations and ignores local patch and word tokens carrying fine-grained spatio-temporal cues.
Approach
Replace cosine similarity with Brownian distance covariance (BDC) computed over all visual tokens (per frame, with temporal attention) and all textual tokens, using Brownian distance correlation (BDCorr) to align modalities, capturing both linear and nonlinear dependencies.
Key results
- Zero-shot with WSE: 60.7% top-1 on HMDB-51, 78.9% on UCF-101, 95.9% on K600 top-5 comparisons vs TC-CLIP 56.0/78.1 (reported gains of 4.7, 1.5, 0.8 points over runners-up).
- Fully supervised K400: 86.5% top-1 with WSE, vs 85.7% for TC-CLIP with WSE.
- Base-to-novel on HMDB-51, UCF-101, SSv2 improves harmonic mean over TC-CLIP by 2.6, 2.0, and 2.1 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace global-token cosine alignment with BDC alignment over visual and textual tokens (ablation) | CLIP ViT-B/16 video action recognition · K600 zero-shot, K400-tiny | accuracy: 71.7±0.9 -> 73.5±0.7 | Helped |
| add a BDC vision classifier (ablation) | BDC-CLIP · K600 zero-shot, K400-tiny | accuracy: 73.5±0.7 -> 73.8±0.8 | Helped |
| use BDC rather than cosine similarity for multimodal alignment (ablation) | CLIP video-language alignment · K600 zero-shot, K400-tiny | accuracy: 71.7±0.9 -> 73.8±0.8 | Helped |
| use BDC-CLIP temporal attention rather than averaging frame-wise BDC matrices (ablation) | BDC-CLIP · K600 zero-shot, K400-tiny | accuracy: 72.5±0.7 -> 73.8±0.8 | Helped |
| use LLM-generated prompts instead of vanilla prompts (ablation) | BDC-CLIP · K600 zero-shot, K400-tiny | accuracy: 73.8±0.8 -> 74.5±1.0 | Helped |
| increase BDC dimension from 128 to 192 (ablation) | BDC-CLIP · K600 zero-shot, K400-tiny | accuracy: 72.7±0.8 -> 73.8±0.9 | Helped |
| vary BDC dimension across 192, 256, 320, and 384 (ablation) | BDC-CLIP · K600 zero-shot, K400-tiny | accuracy: 73.8±0.9; 73.7±0.8; 73.8±0.9; 73.9±0.7 | Mixed |
| retain all visual and text tokens rather than subsampling tokens (ablation) | BDC-CLIP · K600 zero-shot, K400-tiny | accuracy: 73.8±0.8; 73.7±0.7; 73.4±0.7; 73.6±0.7; 73.4±0.7 | Helped |
| apply BDC on both vision and text branches instead of only one branch (ablation) | BDC-CLIP · SSv2 16-shot, K400-tiny | accuracy: 16.0 -> 16.8; 14.3 -> 16.8 | Helped |
| apply token weighting to both branches instead of only one (ablation) | BDC-CLIP · SSv2 2-shot, K400-tiny | accuracy: 7.6 -> 8.9; 8.1 -> 8.9 | Helped |
| fine-tune BDC-CLIP rather than TC-CLIP | CLIP ViT-B/16 · HMDB-51 16-shot | accuracy: 73.0 -> 77.3 | Helped |
| fine-tune BDC-CLIP rather than TC-CLIP | CLIP ViT-B/16 · UCF-101 16-shot | accuracy: 97.3 -> 98.5 | Helped |
Surprising findings
- Cosine similarity between CLIP tokens behaves like Pearson correlation, and cross-modal token relations are non-Gaussian and nonlinear, per the authors' analysis.
Limitations
- Higher GFLOPs and lower throughput than some baselines.
- Currently tailored to video action recognition; not evaluated on retrieval or detection.
- Reliance on LLM (GPT-4o) text augmentation introduces inherited biases.
Experimental setup
Datasets: Kinetics-400 Kinetics-600 HMDB-51 UCF-101 Something-Something v2 ImageNet Aircraft Food101 DTD Cars OxfordPets SUN397
Benchmarks: HMDB-51 UCF-101 Kinetics-600 Kinetics-400 Something-Something v2 ImageNet Aircraft Food101 DTD Cars OxfordPets SUN397
Metrics: top-1 accuracy top-5 accuracy harmonic mean GFLOPS throughput ARI NMI
Baselines: ViFi-CLIP TC-CLIP OST FROSTER Open-VCLIP ActionCLIP X-CLIP A5 Vita-CLIP DiST MoTED BDC-Adapter
Builds on: CLIP ViFi-CLIP DeepBDC BDC-Adapter TC-CLIP CLIP-LoRA
Closest papers by meaning
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.89)
- FROSTER: Frozen CLIP is A Strong Teacher for Open-Vocabulary Action Recognition (ICLR 2024 · similarity 0.87)
- Temporal Alignment-Free Video Matching for Few-shot Action Recognition (CVPR 2025 · similarity 0.86)
- CLIP-KD: An Empirical Study of CLIP Model Distillation (CVPR 2024 · similarity 0.86)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.86)
- Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.