Research map / Image generation / Fast diffusion sampling
Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ATM, an EEG encoder aligned to CLIP image embeddings with a two-stage prior-diffusion plus SDXL/IP-Adapter pipeline for zero-shot EEG visual decoding, retrieval, and image reconstruction, also tested on MEG.
Problem
Visual decoding and image reconstruction from non-invasive EEG lag far behind fMRI in zero-shot settings, limiting EEG use in BCIs.
Approach
A tailored EEG encoder (ATM: channel-wise attention, temporal-spatial convolution, MLP projector) aligns EEG to CLIP embeddings via contrastive loss; a lightweight prior diffusion model maps EEG embeddings to CLIP image priors, and a second stage uses SDXL with IP-Adapter plus a low-level img2img pipeline to generate images.
Key results
- ATM-S achieves 28.64% top-1 and 58.47% top-5 zero-shot EEG retrieval on THINGS-EEG (200-way), versus 8.11%/26.83% for the MLP-only baseline.
- Reconstruction on THINGS-EEG Subject 8 reaches PixCorr 0.160, SSIM 0.345, CLIP two-way 0.786, exceeding the THINGS-MEG reported results on most metrics.
- Reconstructions can be obtained from less than 500 ms of EEG, with image similarity plateauing after about 200-500 ms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add channel-wise attention to MLP + TSConv (ablation) | ATM-S · THINGS-EEG retrieval | Top-1: 21.65± 6.22 -> 28.64±6.39 | Helped |
| Add TSConv to MLP + channel-wise attention (ablation) | ATM-S · THINGS-EEG retrieval | Top-1: 23.73± 7.62 -> 28.64±6.39 | Helped |
| Add TSConv and channel-wise attention to MLP (ablation) | ATM-S · THINGS-EEG retrieval | Top-1: 8.11± 1.74 -> 28.64±6.39 | Helped |
| Add channel-wise attention to MLP + TSConv (ablation) | ATM-E · THINGS-EEG retrieval | Top-1: 17.95±5.95 -> 24.92± 6.00 | Helped |
| Add TSConv to MLP + channel-wise attention (ablation) | ATM-E · THINGS-EEG retrieval | Top-1: 23.73± 7.62 -> 24.92± 6.00 | Helped |
| Use keep-dimension tokens instead of mean tokens (ablation) | ATM channel-wise attention · THINGS-EEG retrieval | Top-1: 7.29 (3.01) -> 28.64 (6.39) | Helped |
| Use keep-dimension tokens instead of flattened tokens (ablation) | ATM channel-wise attention · THINGS-EEG retrieval | Top-1: 14.85 (5.46) -> 28.64 (6.39) | Helped |
| Use linear rather than Conv1d token embedding (ablation) | ATM · THINGS-EEG retrieval | Top-1: 24.81 (7.29) -> 28.64 (6.39) | Helped |
| Use temporal-dimension FFN rather than spatial-dimension FFN (ablation) | ATM · THINGS-EEG retrieval | Top-1: 19.92 (7.74) -> 28.64 (6.39) | Helped |
| Add temporal-dimension FFN rather than omit the FFN (ablation) | ATM · THINGS-EEG retrieval | Top-1: 26.45 (7.73) -> 28.64 (6.39) | Helped |
| Remove positional encoding rather than use sinusoidal encoding (ablation) | ATM · THINGS-EEG retrieval | Top-1: 27.96 (6.54) -> 28.64 (6.39) | No clear effect |
| Place TSConv after the Transformer rather than before it (ablation) | ATM · THINGS-EEG retrieval | Top-1: 14.23 (4.20) -> 28.64 (6.39) | Helped |
Surprising findings
- EEG performed better than MEG on visual decoding tasks in this study, contrary to speech decoding where MEG typically leads.
- Image reconstruction quality was comparable to or better than fMRI on some metrics despite EEG's low spatial resolution.
Limitations
- Cross-subject performance lags within-subject performance substantially.
- Low-level zero-shot reconstruction is unstable and can mislead results, and the semantic caption pipeline is unreliable.
- MEG experiments use only four subjects, so no statistical analysis was performed.
Experimental setup
Datasets: THINGS-EEG THINGS-MEG ImageNet NSD
Benchmarks: THINGS-EEG THINGS-MEG
Metrics: top-1 accuracy top-5 accuracy PixCorr SSIM AlexNet two-way identification Inception CLIP two-way identification SwAV
Baselines: NICE B.D. (Benchetrit et al.) EEGConformer EEGNetv4 MLP MindEye
Builds on: CLIP DALL-E 2 SDXL IP-Adapter ShallowNet
Closest papers by meaning
- What Does the Brain See? Multiview Neural Representation to Demystify the Brain-Visual Alignment (NeurIPS 2026 · similarity 0.93)
- MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding (NeurIPS 2026 · similarity 0.92)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.91)
- EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals (NeurIPS 2024 · similarity 0.90)
- Decoding Natural Images from EEG for Object Recognition (ICLR 2024 · similarity 0.89)
- Neural‑Visual Decoding via Cognitive‑guided Adaptive Blurring and Information‑Constrained Alignment (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.