Research map / Vision-language and multimodal / Cross-modal learning
Cross-modal Multi-task Learning for Multimedia Event Extraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes X-MTL, a cross-modal multi-task framework for multimedia event extraction that uses pseudo-label knowledge distillation and dynamic weight adjustment to jointly learn four tasks without multimedia event annotations.
Problem
Multimedia event extraction needs cross-modal correlations, but no multimedia event annotations exist for training; prior image-text pair methods learn implicit correlations that lack event knowledge and yield limited gains on event-centric tasks.
Approach
Establish cross-modal correlations at the task level by jointly training trigger detection, argument extraction, verb classification and role classification with modality-specific encoders, a modality-shared transformer, textual/visual prompts, and pseudo-label distillation on an image-text corpus combined with dynamic weight adjustment.
Key results
- On M2E2 multimedia event mention, X-MTL reaches 66.2 F1, 4.1 points above UMIE (62.1).
- On multimedia argument role, X-MTL reaches 41.4 F1, 8.2 points above CAMEL (33.2).
- Joint multi-task training with pseudo-label distillation and DWA surpasses single-task models (multimedia event 66.2 vs 62.7 F1; visual event 71.7 vs 66.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use X-MTL instead of CAMEL | M2E2 textual event extraction · M2E2 textual event mention | F1: 55.4 -> 56.6 | Helped |
| Use X-MTL instead of CAMEL | M2E2 textual event extraction · M2E2 textual argument role | F1: 31.1 -> 36.0 | Helped |
| Use X-MTL instead of CAMEL | M2E2 visual event extraction · M2E2 visual event mention | F1: 58.5 -> 71.7 | Helped |
| Use X-MTL instead of CAMEL | M2E2 visual event extraction · M2E2 visual argument role | F1: 24.4 -> 32.2 | Helped |
| Use X-MTL instead of UMIE | M2E2 multimedia event extraction · M2E2 multimedia event mention | F1: 62.1 -> 66.2 | Helped |
| Use X-MTL instead of CAMEL | M2E2 multimedia event extraction · M2E2 multimedia argument role | F1: 33.2 -> 41.4 | Helped |
| Add Dynamic Weight Adjustment (ablation) | X-MTL · M2E2 textual event mention | F1: 55.5 -> 56.6 | Helped |
| Add Dynamic Weight Adjustment (ablation) | X-MTL · M2E2 visual event mention | F1: 67.3 -> 71.7 | Helped |
| Add Dynamic Weight Adjustment (ablation) | X-MTL · M2E2 multimedia event mention | F1: 61.5 -> 66.2 | Helped |
| Add pseudo-label knowledge distillation (ablation) | X-MTL · M2E2 visual event mention | F1: 62.2 -> 71.7 | Helped |
| Add pseudo-label knowledge distillation (ablation) | X-MTL · M2E2 visual argument role | F1: 25.5 -> 32.2 | Helped |
| Add pseudo-label knowledge distillation (ablation) | X-MTL · M2E2 multimedia event mention | F1: 59.8 -> 66.2 | Helped |
Surprising findings
- Naive alternating and joint multi-task training both underperform single-task training, with alternating training causing catastrophic forgetting (visual argument down 9.5 points).
Limitations
- Affected by error propagation from pseudo labels.
- Overall process is relatively complex.
- Visual event extraction depends on an offline object detector (YOLOv8) with confidence threshold 0.8.
Experimental setup
Datasets: ACE 2005 imSitu SWiG VOA Caption M2E2
Benchmarks: M2E2
Metrics: precision recall F1 IoU>0.5 for visual arguments
Baselines: VAD Flat WASE UniCL CAMEL CLIP-Event UMIE ARK Alternating training Joint training
Builds on: CAMEL CLIP BERT ARK Dynamic Weight Adjustment
Closest papers by meaning
- Beyond Grounding: Extracting Fine-Grained Event Hierarchies across Modalities (AAAI 2024 · similarity 0.87)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.86)
- Video Event Extraction with Multi-View Interaction Knowledge Distillation (AAAI 2024 · similarity 0.86)
- EventDance: Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition (CVPR 2024 · similarity 0.85)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.85)
- XKD: Cross-Modal Knowledge Distillation with Domain Alignment for Video Representation Learning (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.