Research map / Video / Video-language understanding
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos
Shaoxiang Chen, Wenhao Jiang, Wei Liu, Yu-Gang Jiang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add interaction on concatenated multimodal features (ablation) | captioning baseline · MSVD | CIDEr: 62.57 -> 66.10 | Improved |
| use pairwise interaction instead of interacting on concatenated features (ablation) | PMI-CAP · MSVD | CIDEr: 66.10 -> 75.30 | Improved |
| use sum fusion instead of concatenation after pairwise interaction (ablation) | PMI-CAP · MSVD | CIDEr: 75.30 -> 78.42 | Improved |
| use learned importance-weighted fusion instead of sum fusion (ablation) | PMI-CAP · MSVD | CIDEr: 78.42 -> 95.17 | Improved |
| add inter-modality interactions to intra-modality only (ablation) | PMI-CAP · MSVD | CIDEr: 88.46 -> 95.17 | Improved |
| add intra-modality interactions to inter-modality only (ablation) | PMI-CAP · MSVD | CIDEr: 70.20 -> 95.17 | Improved |
| add intra-modality interaction alone to concat baseline (ablation) | captioning baseline · MSVD | CIDEr: 62.57 -> 88.46 | Improved |
| inter-modality interaction alone vs baseline (ablation) | captioning baseline · MSVD | CIDEr: 62.57 -> 70.20 | Improved |
| add PMI to the localizer (ablation) | PMI-LOC · Charades-STA | R@1 IoU=0.5: 35.34 -> 37.05 | Improved |
| add video-text local interaction (ablation) | PMI-LOC · Charades-STA | R@1 IoU=0.5: 37.05 -> 38.42 | Improved |
| add l2-norm regularization (ablation) | PMI-LOC · Charades-STA | R@1 IoU=0.5: 38.42 -> 39.73 | Improved |
| add l2-norm regularization (ablation) | PMI-LOC · Charades-STA | R@1 IoU=0.7: 18.63 -> 19.27 | Improved |
Closest papers by meaning
- Unsupervised Domain Adaptative Temporal Sentence Localization with Mutual Information Maximization (AAAI 2024 · similarity 0.88)
- Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval (CVPR 2024 · similarity 0.88)
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.87)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.87)
- Audio-Visual Semantic Graph Network for Audio-Visual Event Localization (CVPR 2025 · similarity 0.87)
- Local-Global Multi-Modal Distillation for Weakly-Supervised Temporal Video Grounding (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
