Research map / Probabilistic and scientific modeling / Variational and generative sampling
Subtractive Mixture Models via Squaring: Representation and Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes squaring subtractive (non-monotonic) mixtures and circuits to represent non-negative functions as probabilistic circuits, with tractable marginalization and an exponential expressiveness separation over monotonic structured-decomposable PCs.
Problem
Learning and inference with subtractive (negative-weight) mixture models is hard because the result must remain non-negative; existing approaches need ad-hoc parameter constraints per component family.
Approach
Represent subtractive mixtures by squaring a linear combination with unconstrained (possibly negative) weights, and generalize this to tensorized circuits via squaring structured-decomposable circuits, yielding squared non-monotonic PCs (NPC^2s) that are non-negative and tractable to renormalize.
Key results
- Squared non-monotonic PCs achieve higher log-likelihoods than monotonic PCs with equal parameters on four of five multivariate datasets.
- Theory: some functions are representable by shallow squared NMMs with polynomial size but require 2^{Omega(|X|)} size for structured-decomposable monotonic PCs.
- NPC^2s distill GPT2 better than monotonic PCs, with log-likelihoods closer to GPT2's.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NPC² with negative parameters instead of a monotonic PC | 2D synthetic continuous density estimation · 2D synthetic continuous data | density estimation performance | Helped |
| Square a monotonic PC (ablation) | 2D synthetic continuous density estimation · 2D synthetic continuous data | density estimation performance | Helped |
| Use NPC² instead of squared monotonic PCs | 2D synthetic continuous density estimation · 2D synthetic continuous data | density estimation performance | Helped |
| Use NPC² with embeddings instead of MPC² with categoricals | discrete synthetic data estimation · Discretized 2D synthetic data | distribution estimation performance | No clear effect |
| Use NPC² instead of monotonic PCs with Binomial input layers | discrete synthetic data estimation · Discretized 2D synthetic data | distribution estimation performance | Helped |
| Use NPC² instead of monotonic PCs | image distribution estimation · Image distributions | distribution estimation performance | No clear effect |
| Use NPC² with Gaussian input layers instead of monotonic PCs | multivariate density estimation · Multivariate continuous data | log-likelihood | Helped |
| Use NPC² instead of squared monotonic PCs | multivariate density estimation · Multivariate continuous data | log-likelihood | Helped |
| Use a binary tree region graph instead of a linear tree region graph (ablation) | NPC² and monotonic PCs · Multivariate continuous data | log-likelihood | Helped |
| Use NPC² with a binary tree region graph instead of TTDE | Gas density estimation · Gas | log-likelihood | Helped |
| Distill GPT2 into NPC² instead of monotonic PCs | GPT2 distillation · Sampled sentences | log-likelihood | Helped |
| Fit NPC² to training sentences | GPT2 distillation · Sampled sentences | log-likelihood | Mixed |
Surprising findings
- Squaring deterministic circuits yields no expressiveness gain over monotonic PCs.
- Binomial inputs with subtraction help, but categorical or embedding inputs show no clear advantage over monotonic PCs with categoricals.
Limitations
- Negative parameters invalidate the probabilistic interpretation of sub-circuits, so latent-variable interpretation and structure learning are not straightforward.
- Regularization is needed since NPC^2s fit training data much better than test data.
- No improvement over monotonic PCs on image distributions in preliminary results.
Experimental setup
Datasets: synthetic 2D density data five multivariate UCI-style datasets including Gas sentences sampled from GPT2
Benchmarks: 2D synthetic density estimation multivariate density estimation benchmarks GPT2 distillation log-likelihood
Metrics: log-likelihood KL divergence
Baselines: RealNVP MADE MAF NSF EiNet-LRS TTDE monotonic PC squared monotonic PC Gaussian full covariance
Builds on: probabilistic circuits RAT-SPN EiNet Born machines TTDE
Closest papers by meaning
- Sum of Squares Circuits (AAAI 2025 · similarity 0.89)
- End-to-end Learning of Gaussian Mixture Priors for Diffusion Sampler (ICLR 2025 · similarity 0.86)
- Efficient Mixture Learning in Black-Box Variational Inference (ICML 2024 · similarity 0.85)
- Sublinear Variational Optimization of Gaussian Mixture Models with Millions to Billions of Parameters (NeurIPS 2026 · similarity 0.84)
- Score-of-Mixture Training: One-Step Generative Model Training Made Simple via Score Estimation of Mixture Distributions (ICML 2025 · similarity 0.84)
- Analysis of Learning a Flow-based Generative Model from Limited Sample Complexity (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.