Research map / Vision-language and multimodal / Cross-modal learning
X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes X-Fi, a transformer-based modality-invariant foundation model with an X-Fusion cross-attention mechanism that handles arbitrary combinations of six sensing modalities for human pose estimation and activity recognition after a single training run.
Problem
Human sensing systems are built for fixed sensor combinations, so adding or removing a modality (camera, LiDAR, mmWave, WiFi, RFID, depth) requires redesigning and retraining the model.
Approach
Modality-specific pretrained encoders feed a shared transformer (cross-modal transformer) whose unified embedding repeatedly attends to per-modality key-value pairs via cross-attention (X-Fusion), so modality-specific information is preserved; training samples modality subsets from a per-modality Binomial existence distribution.
Key results
- On MM-Fi HPE, X-Fi reports averaged MPJPE and PA-MPJPE gains over baselines, e.g. 18.8% MPJPE and 23.8% PA-MPJPE improvement for the five-modality input versus decision-level fusion.
- On XRF55 HAR, X-Fi reaches 89.8% accuracy with mmWave+WiFi+RFID input, 19.2% above the 70.6% decision-level baseline.
- Single-modality HAR is weaker than baseline for WiFi (55.7% vs 77.8%); modality existence probabilities were tuned to raise WiFi single-input accuracy from 29.1% to 55.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use X-Fi rather than feature-level fusion | MM-Fi HPE, I+D+L+R+W · MM-Fi HPE, I+D+L+R+W | MPJPE (mm): 128.9 -> 83.7 | Helped |
| Use X-Fi rather than the baseline | MM-Fi HAR, I+D+L+R · MM-Fi HAR, I+D+L+R | accuracy (%): 72.6 -> 72.2 | Hurt |
| Use X-Fi rather than the baseline | XRF55 HAR, R+W+RF · XRF55 HAR, R+W+RF | accuracy (%): 70.6 -> 89.8 | Helped |
| Use four stacked X-Fusion blocks instead of iterative X-Fusion (ablation) | X-Fusion on MM-Fi HPE · MM-Fi HPE, I+D+L+R+W | MPJPE (mm): 83.7 -> 76.6 | Hurt |
| Use four stacked X-Fusion blocks instead of iterative X-Fusion (ablation) | X-Fusion on MM-Fi HPE · MM-Fi HPE, across modality combinations | MPJPE (mm): +1.27 mm | Hurt |
| Use stacked blocks with the same key-value pairs across layers instead of iterative X-Fusion (ablation) | X-Fusion on MM-Fi HPE · MM-Fi HPE, across modality combinations | MPJPE (mm): 0.93 mm | Hurt |
| Reduce X-Fusion iterations from 4 to 2 (ablation) | X-Fi on XRF55 HAR · XRF55 HAR, W | accuracy (%): 24.2% -> 55.7% | Hurt |
| Remove key-value generators and directly use modality features as key-value pairs (ablation) | X-Fi on XRF55 HAR · XRF55 HAR, W | accuracy (%): 26.8% -> 55.7% | Hurt |
| Add dropout layers with dropout rate 0.3 (ablation) | X-Fi on MM-Fi HAR · MM-Fi HAR, L | accuracy (%): 32.9% -> 12.8% | Hurt |
| Add positional encoding (ablation) | X-Fi on MM-Fi HAR · MM-Fi HAR, L | accuracy (%): 31.9% -> 32.9% | Helped |
| Replace concatenation with averaged feature addition (ablation) | X-Fi on MM-Fi HPE · MM-Fi HPE, I+D+L+R+W | MPJPE (mm): 83.7 -> 92.3 | Hurt |
| Use SGD instead of AdamW (ablation) | X-Fi on MM-Fi HPE · MM-Fi HPE, I+D+L+R+W | MPJPE (mm): 83.7 -> 101.2 | Hurt |
Surprising findings
- Multimodal X-Fi outperforms single-modality inputs for several combinations, and WiFi, weak alone, adds information when combined with other modalities.
- Increasing the transformer depth of the cross-modal transformer from 1 to 4 did not improve performance.
Limitations
- Single-modality performance is poor for some modalities (LiDAR HPE -3.5%, WiFi HAR -22.1%) relative to fixed-combination baselines.
- Feature extractors are pretrained per modality and frozen, so the foundation model does not learn the encoders end to end.
- Evaluated on only two datasets with six modalities; the MM-Fi HAR results also show drops for several single-modality inputs.
Experimental setup
Datasets: MM-Fi XRF55
Benchmarks: MM-Fi XRF55
Metrics: MPJPE PA-MPJPE accuracy
Baselines: Visual Odometry Transformer ImmFusion Meta-Transformer-B16 decision-level fusion feature-level fusion
Builds on: Visual Odometry Transformer Transformer Point Transformer Meta-Transformer
Closest papers by meaning
- MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing (NeurIPS 2023 · similarity 0.88)
- VCR: Learning Valid Contextual Representation for Incomplete Wearable Signals (NeurIPS 2026 · similarity 0.84)
- HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning (NeurIPS 2025 · similarity 0.84)
- Multimodal Sense-Informed Forecasting of 3D Human Motions (CVPR 2024 · similarity 0.83)
- Adaptive Multimodal Fusion: Dynamic Attention Allocation for Intent Recognition (AAAI 2025 · similarity 0.83)
- Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.