Research map / Vision-language and multimodal / Cross-modal learning
Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EGO fusion, which builds similarity relationship graphs from unit-level features, expands them with iterative element-wise graph updates, and fuses them with a learnable operator for video anomaly detection.
Problem
Common feature fusion methods (concatenation, element-wise ops, attention, neural fusion) capture shallow interactions, miss structural relationships, and are costly in high-dimensional space, especially for multi-representational, multi-modal, and multi-domain features.
Approach
Move fusion from feature space to a lower-dimensional homogeneous relationship-graph space: expand each similarity graph by iterative element-wise multiplications, then fuse two graph sequences with a learnable coefficient matrix A, interpreted as a multilinear polynomial in relationship scores.
Key results
- EGO (graph-level) reaches 93.23 AUC on UCSD Ped2, 97.26 on ShanghaiTech, 83.10 on CUHK Avenue and 77.61 on Street Scene, beating MTN fusion on three of four (MTN 92.80, 96.37, 62.06, 71.50 respectively; Avenue and Street gains are larger for EGO except ShT where MTN is higher on ShT-only multi-domai
- On MSAD EGO reports 87.36 AUC, above MTN (I3D) 86.82 and MIST (I3D) 86.65.
- EGO uses 0.091M parameters versus 29.0M for MTN, with training time per epoch of 4.9 s vs 9.7 s on I3D+SwinT in the reported table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EGO instead of feature-level MTN fusion | Video anomaly detection fusion · UCSD Ped2 | AUC: 92.80 -> 93.23 | Helped |
| Use EGO instead of feature-level MTN fusion | Video anomaly detection fusion · ShanghaiTech | AUC: 96.37 -> 97.26 | Helped |
| Use EGO instead of feature-level MTN fusion | Video anomaly detection fusion · CUHK Avenue | AUC: 62.06 -> 83.10 | Helped |
| Use EGO instead of the strongest listed feature-level baseline, Product | Video anomaly detection fusion · Street Scene | AUC: 75.59 -> 77.61 | Helped |
| Use EGO rather than MTN for multi-representational fusion | Feature fusion on ShanghaiTech · ShanghaiTech (I3D + C3D) | AUC: 89.25 -> 87.17 | Hurt |
| Use EGO rather than MTN for multi-representational fusion | Feature fusion on ShanghaiTech · ShanghaiTech (I3D + SwinT; C3D + SwinT) | AUC: 88.80 -> 89.85; 84.45 -> 85.52 | Helped |
| Train jointly on Ped2 and ShanghaiTech with EGO rather than MTN | Multi-domain video anomaly detection · ShanghaiTech test split | AUC: 96.04 -> 95.10 | Hurt |
| Train jointly on Ped2 and ShanghaiTech with EGO rather than MTN | Multi-domain video anomaly detection · Ped2 + ShanghaiTech test split | AUC: 94.60 -> 92.11 | Hurt |
| Use EGO instead of MTN fusion | Large-scale anomaly detection · XD-Violence | AP: 77.17 -> 65.77 | Hurt |
| Use EGO instead of MTN fusion | Large-scale anomaly detection · UCF-Crime | AUC: 85.14 -> 81.71 | Hurt |
| Use EGO rather than MTN for multi-modal fusion | Video anomaly detection fusion · ShanghaiTech (Visual + Text) | AUC: 96.37 -> 97.26 | Helped |
Surprising findings
- Graph-level versions of concatenation, addition, and product often underperform single-modality graph baselines, indicating that naive graph-level fusion is insufficient.
- Raw feature-level single-modality inputs typically outperform their graph representations, reflecting information loss in the graph space.
Limitations
- Evaluated only on binary video anomaly detection; generality to other tasks is not shown.
- Several results are mixed: MTN is better on some settings (e.g., I3D+C3D multi-representation, ShT-only and Ped2+ShT multi-domain), and EGO underperforms on some MSAD scenarios and anomaly types.
Experimental setup
Datasets: UCSD Ped2 ShanghaiTech CUHK Avenue Street Scene XD-Violence UCF-Crime MSAD
Benchmarks: UCSD Ped2 ShanghaiTech CUHK Avenue Street Scene XD-Violence UCF-Crime MSAD
Metrics: AUC AP
Baselines: Concatenation Addition Product MTN fusion I3D visual only Text only MIST UR-DMU MGFN RTFM
Builds on: MTN fusion graph powers Graph Convolutional Networks
Closest papers by meaning
- Beyond Redundancy: Information-aware Unsupervised Multiplex Graph Structure Learning (NeurIPS 2024 · similarity 0.84)
- GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning (CVPR 2026 · similarity 0.84)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.84)
- EGTR: Extracting Graph from Transformer for Scene Graph Generation (CVPR 2024 · similarity 0.83)
- Beyond Spatial-Domain Supervision: A Relation Constrained Space for Multi-Modal Image Fusion (NeurIPS 2026 · similarity 0.83)
- HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.