Research map / Vision-language and multimodal / Cross-modal learning
CognitionCapturer: Decoding Visual Stimuli from Human EEG Signal with Multimodal Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CognitionCapturer, which trains per-modality EEG expert encoders (image, text, depth) with contrastive learning, maps EEG embeddings to CLIP space via a diffusion prior, and reconstructs visual stimuli with SDXL-Turbo and IP-Adapters without fine-tuning the generator.
Problem
Decoding visual stimuli from EEG, where prior work aligns EEG only with image data and so misses beyond-image-modality information in the signal.
Approach
Align EEG with multiple modality embeddings (image, BLIP2-generated text, Depth-Anything depth maps) via modality-specific EEG encoders trained with contrastive loss, then map the EEG embedding to CLIP space with a diffusion prior and feed modality-specific embeddings into SDXL-Turbo with IP-Adapter variants for reconstruction.
Key results
- Image-modality CognitionCapturer reaches 33.30% top-1 and 60.58% top-5 on 200-way zero-shot classification averaged across 10 subjects, versus ATM at 26.1% top-1 and 58.1% top-5.
- Combining all three modalities (upper bound, top-5 fusion) yields 35.64% top-1 and 80.22% top-5.
- Reconstruction with all modalities gives CLIP 0.715 and PixCorr 0.150, with low-level PixCorr matching or exceeding MEG baseline (0.090) but high-level metrics still well below MindEye-fMRI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine image, text, and depth modalities instead of using image modality only (ablation) | CognitionCapturer · Things-EEG 200-way zero-shot | Top-1 accuracy: 33.30 -> 35.64 | Helped |
| combine image, text, and depth modalities instead of using image modality only (ablation) | CognitionCapturer · Things-EEG 200-way zero-shot | Top-5 accuracy: 60.58 -> 80.22 | Helped |
| combine image, text, and depth modalities instead of using image modality only (ablation) | CognitionCapturer · Things-EEG reconstruction | PixCorr: 0.132 -> 0.150 | Helped |
| combine image, text, and depth modalities instead of using image modality only (ablation) | CognitionCapturer · Things-EEG reconstruction | SSIM: 0.321 -> 0.347 | Helped |
| combine image, text, and depth modalities instead of using image modality only (ablation) | CognitionCapturer · Things-EEG reconstruction | AlexNet(2): 0.813 -> 0.754 | Hurt |
| combine image, text, and depth modalities instead of using image modality only (ablation) | CognitionCapturer · Things-EEG reconstruction | CLIP: 0.705 -> 0.715 | Helped |
| add the text modality to image-only decoding (ablation) | CognitionCapturer · Things-EEG reconstruction | PixCorr: 0.102 -> 0.150 | Helped |
| add the depth modality to image-only decoding (ablation) | CognitionCapturer · Things-EEG reconstruction | SSIM: 0.370 -> 0.347 | No clear effect |
| add text and depth modalities to image-only decoding (ablation) | CognitionCapturer · Things-EEG reconstruction | AlexNet(2): 0.813 -> 0.754 | Hurt |
| add text and depth modalities to image-only decoding (ablation) | CognitionCapturer · Things-EEG reconstruction | CLIP: 0.705 -> 0.715 | Helped |
Surprising findings
- The high-level reconstruction metrics for EEG remain well below MEG and fMRI baselines, indicating EEG is harder to decode semantically.
- Text and depth single-modality encoders underperform the image encoder on classification (text 20.88% top-1, depth 27.41% top-1 vs image 33.30%).
Limitations
- Combined multi-modality classification accuracy is an upper bound from top-5 fusion across modalities, not a practical accuracy.
- EEG high-level semantic decoding lags behind MEG and fMRI methods.
- Evaluation is limited to the Things-EEG dataset with 10 subjects.
Experimental setup
Datasets: Things-EEG BLIP-2 generated captions Depth Anything depth maps
Benchmarks: Things-EEG 200-way zero-shot classification Things-EEG image reconstruction metrics
Metrics: Top-1 accuracy Top-5 accuracy PixCorr SSIM AlexNet(2) AlexNet(5) Inception CLIP SwAV
Baselines: BraVL NICE ATM META-MEG MindEye-fMRI
Builds on: CLIP IP-Adapter SDXL-Turbo infoNCE ATM
Closest papers by meaning
- Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion (NeurIPS 2024 · similarity 0.89)
- MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding (NeurIPS 2026 · similarity 0.89)
- EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals (NeurIPS 2024 · similarity 0.88)
- What Does the Brain See? Multiview Neural Representation to Demystify the Brain-Visual Alignment (NeurIPS 2026 · similarity 0.88)
- Neural‑Visual Decoding via Cognitive‑guided Adaptive Blurring and Information‑Constrained Alignment (NeurIPS 2026 · similarity 0.87)
- Decoding Natural Images from EEG for Object Recognition (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.