Research map / Vision-language and multimodal / Cross-modal learning
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings
Michael Wray, Diane Larlus, Gabriela Csurka, Dima Damen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn separate verb and noun embeddings and combine with a joint PoS-agnostic embedding (JPoSE) instead of one caption embedding | MMEN · EPIC seen, vt | mAP: 14.0 -> 23.2 | Improved |
| same, JPoSE vs MMEN (Caption) | MMEN · EPIC unseen, tv | mAP: 7.7 -> 10.2 | Improved |
| add the final joint PoS-agnostic embedding on top of concatenated verb and noun MMENs (ablation) | MMEN ([Verb, Noun]) · EPIC seen, vt | mAP: 18.7 -> 23.2 | Improved |
| embed verb and noun separately and concatenate instead of a single caption embedding (ablation) | MMEN · EPIC seen, vt | mAP: 14.0 -> 18.7 | Improved |
| train PoS-MMENs jointly with PoS-agnostic loss rather than independently (concatenation) (ablation) | JPoSE · EPIC seen, vt | mAP: 21.5 -> 22.7 | Improved |
| use concatenation instead of sum encoding (joint training) (ablation) | JPoSE · EPIC seen, vt | mAP: 21.0 -> 22.7 | Improved |
| learn final embeddings f-hat and g-hat instead of identity (ablation) | JPoSE · EPIC seen, vt | mAP: 22.7 -> 23.2 | Improved |
| train on open vocabulary captions instead of closed vocabulary classes (ablation) | JPoSE · EPIC seen, vt | mAP: 18.0 -> 23.2 | Improved |
| use JPoSE instead of MMEN (Caption) for zero-shot verbs | MMEN · EPIC ZSV, vt | mAP: 5.77 -> 7.50 | Improved |
| use JPoSE instead of MMEN (Caption) for zero-shot nouns | MMEN · EPIC ZSN, vt | mAP: 4.17 -> 7.68 | Improved |
| replace the MLP caption encoder with a GRU (Caption RNN) (ablation) | MMEN (Caption) · EPIC seen, vt | mAP: 14.0 -> 10.3 | Mixed |
| disentangle nouns from caption (JPoSE Caption\Noun, Noun) on MSR-VTT | MMEN (Caption) · MSR-VTT text-to-video | R@1: 13.8 -> 14.3 | Improved |
Closest papers by meaning
- Storyboard-guided Alignment for Fine-grained Video Action Recognition (NeurIPS 2025 · similarity 0.87)
- Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning (AAAI 2024 · similarity 0.86)
- From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos (NeurIPS 2025 · similarity 0.86)
- SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization (AAAI 2025 · similarity 0.86)
- Diffusion-Inspired Truncated Sampler for Text-Video Retrieval (NeurIPS 2024 · similarity 0.86)
- Tri-Modal Motion Retrieval by Learning a Joint Embedding Space (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
