Research map / Probabilistic and scientific modeling / Variational and generative sampling
Autoencoding Conditional Neural Processes for Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PPS-VAE, a variational autoencoder that learns which pixels (context set) a conditional neural process should observe, and evaluates the learned pixel sets for image modelling and classification probing.
Problem
Context-set selection for conditional neural processes (CNPs) in image completion is typically random or based on simple statistics, and it is unclear whether good context pixels are meaningful representations of image content.
Approach
Cast the CNP context set as latent variables: an amortised variational posterior over pixel locations (autoregressive, CNN-based) plus an abstractive latent, trained jointly with the CNP decoder so that the selected pixels form a partial pixel specification (PPS) of the image.
Key results
- PPS-VAE attains the best log marginal likelihood estimate on all four datasets (e.g. CelA 14210 vs 12164 for PPS-CAE and 12064 for ConvCNP at M=128).
- On CLEVR classification, PPS-VAE with post-hoc tiles reaches 94.62 F1-macro, above the Image baseline's 91.90 and VQ-VAE's 75.91.
- Increasing context size at inference (128->256) without retraining raises FER2013 F1 from 46.72 to 51.61, the best among encoder-based models on that dataset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn instance-specific context sets instead of randomly sampling context points | PPS-VAE vs ConvCNP · FER2013, CelebA, CLEVR, t-ImageNet | log marginal likelihood: 4472 -> 4951; 12064 -> 14210; 15981 -> 16611; 15793 -> 16324 | Mixed |
| learn a posterior over context sets rather than only a global prior | PPS-VAE vs PPS-CAE · FER2013, CelebA, CLEVR, t-ImageNet | log marginal likelihood: 4471 -> 4951; 12162 -> 14210; 16089 -> 16611; 15832 -> 16324 | Helped |
| increase context size from M=32 to M=128 (ablation) | PPS-VAE · FER2013, CelebA, CLEVR, t-ImageNet | log marginal likelihood: 4111 -> 4951; 11569 -> 14210; 16529 -> 16611; 15645 -> 16324 | Helped |
| increase context size from M=32 to M=128 (ablation) | PPS-VAE · CelebA A:13, CelebA A:20, CelebA A:25, FER2013, CLEVR, t-ImageNet | F1-macro: 57.62 ± 0.85 -> 69.00 ± 0.38; 88.28 ± 0.07 -> 94.86 ± 0.12; 57.57 ± 0.88 -> 62.13 ± 0.50; 29.19 ± 3.73 -> 46.72 ± 0.62; | Helped |
| increase inference context size from 128 to 256 (ablation) | PPS-VAE · CelebA A:13, CelebA A:20, CelebA A:25, FER2013, CLEVR, t-ImageNet | F1-macro: 69.00 ± 0.38 -> 69.94 ± 0.50; 94.86 ± 0.12 -> 95.70 ± 0.07; 62.13 ± 0.50 -> 62.02 ± 0.50; 46.72 ± 0.62 -> 51.61 ± 0.57; | Helped |
| add post-hoc 3x3 tiles around selected context points (ablation) | PPS-VAE · CelebA A:13, CelebA A:20, CelebA A:25, FER2013, CLEVR, t-ImageNet | F1-macro: 69.00 ± 0.38 -> 70.94 ± 0.09; 94.86 ± 0.12 -> 96.21 ± 0.04; 62.13 ± 0.50 -> 62.94 ± 0.10; 46.72 ± 0.62 -> 49.38 ± 0.39; | Mixed |
| use an independent instead of autoregressive context-location encoder (ablation) | PPS-VAE · FashionMNIST, CIFAR-10 | F1-macro: 88.0 ± 0.0 -> 86.0 ± 0.0; 76.7 ± 0.5 -> 68.0 ± 0.0 | Hurt |
| replace PPS-VAE's learned representation with its abstractive latent a (ablation) | PPS-VAE · CelebA A:13, CelebA A:20, CelebA A:25, FER2013, CLEVR, t-ImageNet | F1-macro: 59.04 ± 0.66 -> 54.26 ± 0.42; 86.26 ± 0.12 -> 83.52 ± 0.09; 58.60 ± 0.27 -> 55.42 ± 0.24; 36.06 ± 0.34 -> 20.83 ± 0.13; | Hurt |
| train with a convolutional rather than MLP parameterization (ablation) | PPS-VAE context encoder and CNP · FashionMNIST | spatial meaning of selected points | Helped |
| pretrain on t-ImageNet rather than CelebA or CLEVR | PPS-VAE, FSQ-VAE, and VQ-VAE · Out-of-distribution classification | F1-macro | Mixed |
Surprising findings
- Probing the learned context set (about 3% of pixels) on CLEVR outperforms classifiers trained on the full original image (90.21 vs 91.90 is close; post-hoc tiles 94.62 exceed the image baseline).
- An MLP-based CNP decoder biased the learned context points toward image edges, motivating the CNN-based inductive bias.
Limitations
- Human-level interpretability of the learned context set is limited; the analysis is observational.
- The number of context points M is fixed in advance rather than learned per image.
- Zero-shot reconstruction is poor when pre-trained on CLEVR, and out-of-distribution classification degradation depends on the pre-training dataset.
Experimental setup
Datasets: FER2013 CelebA CLEVR Tiny ImageNet FashionMNIST CIFAR-10
Benchmarks: FER2013 CelebA (A:13, A:20, A:25) CLEVR Tiny ImageNet FashionMNIST CIFAR-10
Metrics: log marginal likelihood estimate F1-macro
Baselines: VQ-VAE FSQ-VAE PPS-CAE ConvCNP PPS-RAND Image baseline VAE
Builds on: conditional neural processes ConvCNP Concrete Autoencoder VQ-VAE Gumbel-Softmax
Closest papers by meaning
- Spatially Informed Autoencoders for Interpretable Visual Representation Learning (ICLR 2026 · similarity 0.87)
- Gaussian Process Prior Variational Autoencoders (NeurIPS 2018 · similarity 0.87)
- Autoregressive Conditional Neural Processes (ICLR 2023 · similarity 0.86)
- Conditional Neural Processes (ICML 2018 · similarity 0.86)
- Conditional Image Generation by Conditioning Variational Auto-Encoders (ICLR 2022 · similarity 0.86)
- Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.