Research map / Image generation / Autoregressive and latent generation
Image-to-Brain Signal Generation for Visual Prosthesis with CLIP Guided Multimodal Diffusion Models
Ganxi Xu, Zhao-Rong Lai, Yuting Tang, Yonghao Song, Guoxu Zhou, Boyu Wang, Jian Zhu, Jinyi Long
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CLIP text embeddings from LLM captions (ablation) | DiT image-to-brain model · THINGS-EEG2 | MSE: 0.117 -> 0.109 | Improved |
| add CLIP text embeddings from LLM captions (ablation) | DiT image-to-brain model · THINGS-MEG | MSE: 0.677 -> 0.663 | Improved |
| add brain region embeddings (ablation) | DiT image-to-brain model · THINGS-EEG2 | Pearson: 0.383 -> 0.425 | Improved |
| add brain region embeddings (ablation) | DiT image-to-brain model · THINGS-MEG | Pearson: 0.337 -> 0.379 | Improved |
| add temporal embeddings (ablation) | DiT image-to-brain model · THINGS-EEG2 | Pearson: 0.395 -> 0.425 | Improved |
| add temporal embeddings (ablation) | DiT image-to-brain model · THINGS-MEG | Pearson: 0.349 -> 0.379 | Improved |
| use Qwen2-VL-2B instead of VisualGLM for captions (ablation) | DiT image-to-brain model · THINGS-EEG2 | MSE: 0.147 -> 0.109 | Improved |
| use Qwen2-VL-2B instead of MiniGPT-4 for captions (ablation) | DiT image-to-brain model · THINGS-MEG | MSE: 0.689 -> 0.663 | Improved |
| keep occipital channels (vs removing them) (ablation) | DiT image-to-brain model · THINGS-EEG2 | MSE: 0.200 -> 0.109 | Improved |
| keep occipital channels (vs removing them) (ablation) | DiT image-to-brain model · THINGS-MEG | Pearson: 0.295 -> 0.379 | Improved |
| use DiT with cross-attention instead of SynBrain | image-to-brain generation · THINGS-EEG2 | Pearson: 0.366 -> 0.425 | Improved |
| test on different subjects than training (cross-subject) | Our work · THINGS-MEG | Pearson: 0.379 -> 0.082 | Hurt |
Closest papers by meaning
- Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion (NeurIPS 2024 · similarity 0.90)
- CognitionCapturer: Decoding Visual Stimuli from Human EEG Signal with Multimodal Information (AAAI 2025 · similarity 0.88)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.87)
- MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding (NeurIPS 2026 · similarity 0.87)
- What Does the Brain See? Multiview Neural Representation to Demystify the Brain-Visual Alignment (NeurIPS 2026 · similarity 0.86)
- MindCustomer: Multi-Context Image Generation Blended with Brain Signal (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
