Research map / Vision-language and multimodal / CLIP and zero-shot
X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes X-MIC, a late-fusion cross-modal adapter that turns each egocentric video into a vector added to frozen CLIP text embeddings, evaluated for cross-dataset noun/verb action generalization on Ego4D, Epic-Kitchens and EGTEA.
Problem
Adapting vision-language models to egocentric video action recognition generalizes poorly across datasets (e.g., zero-shot CLIP reaches only 8.7% noun and 4.3% verb accuracy on Epic-Kitchens), and existing adaptation methods overfit to shared classes.
Approach
Instead of tuning the text encoder or fusing features after the encoders, an ego-spatio-temporal adapter maps frozen per-frame CLIP (or DINO) features plus hand-crop features to a video-specific X-MIC vector that is added to every frozen class text embedding, yielding video-conditioned classifiers without backpropagating through either encoder.
Key results
- On models trained on Ego4D, X-MIC reaches 33.54% noun and 28.93% verb accuracy on Ego4D and 15.35% / 26.48% on Epic-Kitchens, outperforming Vita-CLIP and A5 in harmonic mean.
- X-MIC+DINO improves the Ego4D-trained noun harmonic mean to 24.80 versus 16.11 for Vita-CLIP.
- On EGTEA zero-shot from Ego4D, X-MIC reaches 29.21% noun and 31.41% verb accuracy, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DINO as Visual Encoder II | X-MIC · Ego4D → Epic-Kitchens nouns, harmonic mean | harmonic mean: 21.06 -> 24.80 | Helped |
| Add DINO as Visual Encoder II | X-MIC · Ego4D → Epic-Kitchens verbs, harmonic mean | harmonic mean: 27.65 -> 28.86 | Helped |
| Add hand crops alongside full frames in ego-spatial-temporal attention (ablation) | X-MIC · Ego4D → Epic-Kitchens nouns | harmonic mean: 19.61 -> 21.06 | Helped |
| Add hand crops alongside full frames in ego-spatial-temporal attention (ablation) | X-MIC · Ego4D → Epic-Kitchens verbs | harmonic mean: 25.51 -> 27.65 | Helped |
| Use CLIP ViT-L/14 instead of ViT-B/16 (ablation) | X-MIC · Ego4D → Epic-Kitchens nouns | harmonic mean: 21.06 -> 26.97 | Helped |
| Use CLIP ViT-L/14 instead of ViT-B/16 (ablation) | X-MIC · Ego4D → Epic-Kitchens verbs | harmonic mean: 27.65 -> 28.52 | Helped |
| Use LaViLa ViT-L/14 as the backbone (ablation) | X-MIC · Ego4D → Epic-Kitchens nouns | harmonic mean: 21.06 -> 35.08 | Helped |
| Use LaViLa ViT-L/14 as the backbone (ablation) | X-MIC · Ego4D → Epic-Kitchens verbs | harmonic mean: 27.65 -> 16.37 | Hurt |
| Remove feature normalization (ablation) | X-MIC · Ego4D → Epic-Kitchens nouns | harmonic mean: 21.06 -> 19.92 | Hurt |
| Normalize X-MIC and text features instead of input features (ablation) | X-MIC · Ego4D → Epic-Kitchens nouns | harmonic mean: 21.06 -> 20.19 | Hurt |
| Normalize input and X-MIC features (ablation) | X-MIC · Ego4D → Epic-Kitchens nouns | harmonic mean: 21.06 -> 13.83 | Hurt |
| Normalize input and text features (ablation) | X-MIC · Ego4D → Epic-Kitchens nouns | harmonic mean: 21.06 -> 11.71 | Hurt |
Surprising findings
- CoOp shows robust zero-shot novel-class generalization even without a temporal module, while video adapters such as A5 and Vita-CLIP overfit to shared classes.
- Verb recognition is sensitive to prompt context and performs best with no additional context, and LaViLa's egocentric pretraining improved nouns but not within-dataset verbs.
Limitations
- Focuses solely on video classification and does not cover text-to-video retrieval.
- Verb generalization remains weaker than noun generalization, attributed to object-centric backbone pretraining.
Experimental setup
Datasets: Ego4D Epic-Kitchens-100 EGTEA
Benchmarks: Ego4D FHO Epic-Kitchens-100 EGTEA
Metrics: top-1 accuracy harmonic mean
Baselines: Zero-shot CLIP CoOp CoCoOp CLIP-Adapter A5 Vita-CLIP
Builds on: CoOp CoCoOp CLIP-Adapter A5 Vita-CLIP
Closest papers by meaning
- Retrieval-Augmented Egocentric Video Captioning (CVPR 2024 · similarity 0.88)
- Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos (ICLR 2025 · similarity 0.88)
- SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos (CVPR 2024 · similarity 0.87)
- MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA (ICLR 2025 · similarity 0.86)
- Viewpoint Rosetta Stone: Unlocking Unpaired Ego-Exo Videos for View-invariant Representation Learning (CVPR 2025 · similarity 0.86)
- Align Before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.