Research map / Video / Action recognition
Beyond Image Classification: A Video Benchmark and Dual-Branch Hybrid Discrimination Framework for Compositional Zero-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces C-EgoExo, a video-based compositional action recognition benchmark built from EgoExo-4D, and a dual-branch hybrid discrimination (DHD) framework with a copula-based orthogonal decoding loss for compositional zero-shot learning.
Problem
Compositional zero-shot learning is evaluated almost exclusively on image attribute-object classification, which may cause overfitting to image tasks and fail to measure compositional generalization across video and action-object compositions.
Approach
A dual-branch design decodes condition (verb/attribute) and object primitives in two orders (condition-first and object-first), links them via a contextual dependency encoder with cross-attention, and uses a copula-entropy-based orthogonal loss to reduce cross-branch redundancy and conditional variance.
Key results
- On C-EgoExo, DHD reaches AUC 4.8 and HM 18.3, the best among compared methods (C2C AUC 4.4, HM 17.3).
- On UT-Zappos, DHD reaches AUC 36.4 and HM 48.1 (vs. C2C 30.3 AUC); on CGQA, DHD AUC 9.5 vs. C2C 8.2.
- Ablation on C-EgoExo: adding copula orthogonal loss and contextual dependency encoder raises HM from 15.1 to 18.3 and AUC from 3.6 to 4.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DHD instead of C2C | CZSL models · C-EgoExo | HM: 17.3 -> 18.3 | Helped |
| Add the object-first branch to the condition-first branch (ablation) | DHD · C-EgoExo | HM: 15.1 -> 17.5 | Helped |
| Add contextual dependency encoding (ablation) | DHD with both branches · C-EgoExo | HM: 17.5 -> 18.3 | Helped |
| Add Lobj to Lcon (ablation) | DHD loss · C-EgoExo | HM: 15.8 -> 16.3 | Helped |
| Add Lccd (ablation) | DHD loss · C-EgoExo | AUC: +9.8% | Helped |
| Add Locd (ablation) | DHD loss · C-EgoExo | HM: 16.9 -> 17.7 | Helped |
| Add Lort (ablation) | DHD loss · C-EgoExo | AUC: +6.7% | Helped |
| Increase loss balance parameter β (ablation) | DHD · C-EgoExo | AUC: approximately 16.7% | Mixed |
| Tune contextual loss parameter σ (ablation) | DHD · C-EgoExo | AUC: up to 25.0% | Mixed |
| Use object-only rather than condition-only inference (ablation) | DHD cross-domain evaluation · C-EgoExo trained on Sth-com | HM: 12.9 -> 13.6 | Helped |
Surprising findings
- Performance on image-based CZSL benchmarks correlates weakly with performance on the video benchmark; e.g., CGE drops from near-top AUC on UT-Zappos to much lower relative AUC on C-EgoExo.
- Object-only branch alone performs nearly as well as the full model in cross-domain evaluation, with condition-only variant weaker.
Limitations
- Image-based gains are limited because the temporal module adds little for static images (AUC gains over C2C only 20.1% and 15.9% on UT-Zappos and CGQA).
- Cross-domain transfer from Sth-com to C-EgoExo reaches only about 62.7% of in-domain performance.
- Annotation pipeline mixes automatic parsing (Stanza, SpaCy) with manual annotation of 33,510 descriptions due to inconsistent tool outputs.
Experimental setup
Datasets: C-EgoExo EgoExo-4D UT-Zappos CGQA Sth-com
Benchmarks: C-EgoExo UT-Zappos CGQA
Metrics: best seen accuracy (S) best unseen accuracy (U) harmonic mean (HM) area under curve (AUC)
Baselines: TMN Compcos CGE OADis ADE CAN C2C
Builds on: Compcos ADE C2C TSM
Closest papers by meaning
- Revealing the Proximate Long-Tail Distribution in Compositional Zero-Shot Learning (AAAI 2024 · similarity 0.89)
- Retrieval-Augmented Primitive Representations for Compositional Zero-Shot Learning (AAAI 2024 · similarity 0.89)
- A Dynamic Learning Method towards Realistic Compositional Zero-Shot Learning (AAAI 2024 · similarity 0.89)
- Context-based and Diversity-driven Specificity in Compositional Zero-Shot Learning (CVPR 2024 · similarity 0.88)
- ProCC: Progressive Cross-Primitive Compatibility for Open-World Compositional Zero-Shot Learning (AAAI 2024 · similarity 0.88)
- Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.