Research map / Vision-language and multimodal / Human and embodied modeling
Image Fusion via Vision-Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FILM, a image fusion paradigm that generates text descriptions of source images with BLIP2, GRIT, Segment Anything and ChatGPT, fuses them in text space, and uses cross-attention to guide visual fusion, and releases a VLF vision-language dataset.
Problem
Existing image fusion methods rely on pixel-level and visual-semantic features and overlook deeper text-level semantic information that could guide fusion across infrared-visible, medical, multi-exposure, and multi-focus tasks.
Approach
Derive explicit textual descriptions of source images via a vision-language pipeline (captions, dense captions, semantic masks fed to ChatGPT), fuse the text features with a frozen BLIP2 text encoder, and use the fused text as the query in cross-attention to guide visual feature fusion and decoding.
Key results
- On MSRS infrared-visible fusion, FILM reports EN 6.72, SD 43.17, SF 11.70, AG 3.84, VIF 1.06, Qabf 0.73 (best on most metrics).
- On Harvard medical fusion (zero-shot from IVF model), FILM reports EN 4.74, SF 23.36, AG 6.19, VIF 0.78, Qabf 0.76, best on most metrics.
- Ablation on RoadScene shows removing text guidance drops EN from 7.43 to 7.16 and VIF from 0.69 to 0.51 relative to full FILM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add text-guided feature extraction and fusion (ablation) | FILM · RoadScene test | Qabf: 0.48 -> 0.62 | Helped |
| Use image caption, dense caption, and segment-mask prompts rather than bypassing these prompts (ablation) | FILM · RoadScene test | Qabf: 0.51 -> 0.62 | Helped |
| Add dense-caption and segment-mask prompts (ablation) | FILM · RoadScene test | Qabf: 0.53 -> 0.62 | Helped |
| Add segment-mask prompts (ablation) | FILM · RoadScene test | Qabf: 0.55 -> 0.62 | Helped |
| Use GPT to integrate the image captions into descriptions (ablation) | FILM · RoadScene test | Qabf: 0.53 -> 0.62 | Helped |
Surprising findings
- Medical fusion results from the IVF-trained model without fine-tuning exceed those of models fine-tuned on medical data, per the authors.
- Replacing ChatGPT summarization with simple concatenation of captions lowers fusion metrics, though it is still reported as a component.
Limitations
- Relies on ChatGPT API calls to generate descriptions, which is costly and not reproducible in the pipeline.
- Evaluation uses only classic non-learned fusion metrics (EN, SD, SF, AG, VIF, Qabf) without semantic or downstream task evaluation.
- Paper notes the high computational cost of invoking vision-language components, motivating the released dataset.
Experimental setup
Datasets: MSRS M3FD RoadScene Harvard Medical SICE MEFB RealMFF Lytro
Benchmarks: MSRS M3FD RoadScene Harvard Medical SICE MEFB RealMFF Lytro
Metrics: EN SD SF AG VIF Qabf
Baselines: SDNet TarDAL DeFusion MetaFusion CDDFuse LRRNet MURF DDFM SegMIF DIFNet U2Fusion MATR
Builds on: BLIP-2 Restormer Segment Anything GRIT
Closest papers by meaning
- 3D Vision-Language Gaussian Splatting (ICLR 2025 · similarity 0.88)
- DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.88)
- Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion (CVPR 2024 · similarity 0.87)
- Remodeling Semantic Relationships in Vision-Language Fine-Tuning (AAAI 2026 · similarity 0.87)
- One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion (CVPR 2025 · similarity 0.87)
- Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.