Research map / Image generation / Super-resolution
Learning Large-Factor EM Image Super-Resolution with Generative Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a VQGAN-Indexer generative framework with multi-scale prior fusion and multi-image alignment/fusion modules for large-factor (8× and 16×) electron microscopy image super-resolution.
Problem
Large-factor (8× and 16×) super-resolution of electron microscopy images to reduce acquisition time and expand field of view, where existing single-image and video SR methods fail to produce faithful, segmentation-friendly results.
Approach
Learn a discrete HR cell-specific codebook with VQGAN and train a latent vector indexer that maps LR EM images to codebook latent vectors; fuse the resulting multi-scale generative reference features via a learned-mask MPF module, then align and fuse neighboring sections with POD (SPyNet + deformable conv) and 3DA modules.
Key results
- On CREMI C at 8×, the method reaches 29.70 PSNR / 0.2291 DISTS and VOI 2.94 (Superhuman) vs 3.35 for SwinIR, with best LPIPS (0.3568) and DISTS.
- At 16× on CREMI C, the method achieves LPIPS 0.4790 and DISTS 0.2927 (best perceptual), with Superhuman VOI 2.96 vs 2.94-ish for the best baseline ranking second in VOI at 16× for MALA.
- Ablation on Table 5 shows adding MPF, POD and 3DA together lowers Superhuman VOI from 4.59 to 2.96 and LPIPS from 0.5168 to 0.4790.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use generative priors with MPF, POD, and 3DA instead of the residual-block control (ablation) | Large-factor EMSR framework · CREMI C 16× | LPIPS: 0.5168 -> 0.4790 | Helped |
| Add POD multi-image feature alignment (ablation) | EMSR framework with MPF and 3DA · CREMI C 16× | LPIPS: 0.4858 -> 0.4790 | Helped |
| Add 3DA multi-image feature fusion (ablation) | EMSR framework with MPF and POD · CREMI C 16× | LPIPS: 0.4815 -> 0.4790 | Helped |
| Add MPF generative-prior fusion (ablation) | EMSR framework with POD and 3DA · CREMI C 16× | LPIPS: 0.5063 -> 0.4790 | Helped |
| Add POD multi-image feature alignment (ablation) | EMSR framework with MPF and 3DA · CREMI C 16× | PSNR: 23.5422 -> 23.6767 | Helped |
| Add 3DA multi-image feature fusion (ablation) | EMSR framework with MPF and POD · CREMI C 16× | PSNR: 23.5548 -> 23.6767 | Helped |
| Add MPF generative-prior fusion (ablation) | EMSR framework with POD and 3DA · CREMI C 16× | PSNR: 23.5287 -> 23.7354 | Helped |
| Set compression patch size to 16 rather than 8 (ablation) | VQGAN prior exploration and indexer · prior indexing | PSNR: 35.0731 -> 28.6876; 20.1212 -> 21.7423 | Mixed |
| Set compression patch size to 16 rather than 32 (ablation) | VQGAN prior exploration and indexer · prior indexing | PSNR: 28.6876 -> 23.1879; 21.7423 -> 21.5291 | Mixed |
| Increase codebook size from 1024 to 2048 latent vectors (ablation) | VQGAN prior exploration · prior exploration | PSNR: 28.6876 -> 28.7159 | Helped |
| Reduce codebook size from 1024 to 512 latent vectors (ablation) | VQGAN prior exploration · prior exploration | PSNR: 28.6876 -> 28.3165 | Hurt |
| Increase latent-vector dimension from 512 to 1024 (ablation) | VQGAN prior exploration · prior exploration | PSNR: 28.6876 -> 28.7574 | Helped |
Surprising findings
- PSNR and SSIM do not align with perceptual quality or segmentation; the Stage III fidelity loss lowered PSNR while improving LPIPS and VOI.
- Real-ESRGAN, despite adversarial training, introduces noise-like artifacts that hurt results.
Limitations
- PSNR and SSIM are not the best; the method is second-best in fidelity metrics behind EDVR.
- Relies on a pretrained prior from a specific EM dataset (FAFB subset) and per-dataset three-stage training.
- Segmentation evaluation depends on fine-tuned Superhuman and MALA networks.
Experimental setup
Datasets: FAFB CREMI Kasthuri15 AC3 AC4
Benchmarks: CREMI C AC4
Metrics: PSNR SSIM LPIPS DISTS VOI ARAND
Baselines: Bicubic RCAN SwinIR BSRN Real-ESRGAN EDVR BasicVSR
Builds on: VQGAN EDVR BasicVSR SPyNet contextual loss
Closest papers by meaning
- Joint EM Image Super-Resolution and Segmentation with Semantic and Structural Priors (NeurIPS 2026 · similarity 0.88)
- $\epsilon$-Seg: Sparsely Supervised Semantic Segmentation of Microscopy Data (NeurIPS 2025 · similarity 0.83)
- A Gaussian Filter-Based 3D Registration Method for Series Section Electron Microscopy (AAAI 2025 · similarity 0.83)
- CryoGEM: Physics-Informed Generative Cryo-Electron Microscopy (NeurIPS 2024 · similarity 0.82)
- From Pixels to Views: Learning Angular-Aware and Physics-Consistent Representations for Light Field Microscopy (NeurIPS 2025 · similarity 0.82)
- CryoFM: A Flow-based Foundation Model for Cryo-EM Densities (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.