Research map / Probabilistic and scientific modeling / Molecules and proteins
S$\Omega$I: Score-based O-INFORMATION Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SΩI, a score-based estimator of O-information (and its gradient) that uses a single amortized denoising network to estimate total correlation and dual total correlation without Gaussian or discrete assumptions.
Problem
Estimating O-information, a multivariate synergy-redundancy measure, is limited to discrete or Gaussian distributions and existing neural approaches scale poorly with the number of variables.
Approach
Express total correlation and dual total correlation as integrals over time of denoising-score differences under a VP-SDE, and amortize all required denoisers (joint, marginal, conditional) in one denoising network, so O-information is estimated with a single model instead of many pairwise MI estimators.
Key results
- On the redundancy benchmark with N=10 variables, SΩI stays accurate as variable dimension grows from 5 to 20, while pairwise MI-based baselines degrade considerably at higher dimension.
- On the synergy and mixed benchmarks, SΩI gives consistent estimates and correctly reports negative O-information for synergy-dominant systems, whereas baselines perform poorly.
- On mouse visual cortex recordings, O-information is higher for change flashes than no-change flashes, consistent with prior PID-based findings but without partitioning regions into sources and target.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SOI instead of pairwise neural MI-estimator baselines | O-information estimation · Synthetic redundancy benchmark | O-information estimation accuracy | Helped |
| Increase variable dimension | Pairwise neural MI-estimator baselines · Synthetic redundancy benchmark | O-information estimation accuracy | Hurt |
| Use SOI instead of pairwise neural MI-estimator baselines | O-information estimation · Synthetic synergy benchmark | O-information estimation accuracy | Helped |
| Use SOI instead of pairwise neural MI-estimator baselines | O-information estimation · Synthetic mixed-interaction benchmark | O-information estimation accuracy | Helped |
| Apply a half-cube transformation to the benchmark distributions | SOI and neural MI-estimator baselines · Half-cube transformed synthetic benchmarks | O-information estimation accuracy | Mixed |
| Use a transformer-based denoising network instead of the MLP network (ablation) | SOI gradient estimation · Synthetic mixed-interaction benchmark | O-information gradient estimation accuracy | Helped |
| Analyze change rather than no-change stimulus flashes | SOI on visual cortex activity · Visual Behavior Neuropixels, 3 brain regions | O-information | Helped |
| Extend the analysis from 3 to 6 visual cortex regions | SOI on visual cortex activity · Visual Behavior Neuropixels, change and no-change flashes | O-information pattern across stimulus conditions | No clear effect |
Surprising findings
- The gradient of O-information estimates show bias, which the authors attribute to the large number of denoising score functions learned by one network.
Limitations
- Gradient of O-information estimation shows bias, likely due to the many denoising score functions the single network must fit.
- On synergy benchmark with half-cube transformation, SΩI shows bias at high synergy with very low O-information values.
- Baselines were only trained with a fixed budget and some diverged, so comparison depends on tuning choices.
Experimental setup
Datasets: synthetic Gaussian redundancy, synergy and mixed systems Allen Brain Observatory Visual Behavior Neuropixels dataset
Benchmarks: synthetic Gaussian redundancy benchmark synthetic synergy benchmark synthetic mixed benchmark Visual Behavior Neuropixels mouse recordings
Metrics: O-information estimate vs analytic ground truth gradient of O-information
Baselines: MINE-based O-information NWJ-based O-information InfoNCE-based O-information CLUB-based O-information
Builds on: O-information score-based generative modeling (VP-SDE) denoising score matching neural mutual information estimators (MINE, NWJ, InfoNCE, CLUB) score-based KL divergence estimation
Closest papers by meaning
- A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets (NeurIPS 2024 · similarity 0.83)
- Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlations (NeurIPS 2025 · similarity 0.83)
- Approximating mutual information of high-dimensional variables using learned representations (NeurIPS 2024 · similarity 0.82)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.82)
- Information-Theoretic Discrete Diffusion (NeurIPS 2025 · similarity 0.82)
- On the sample complexity of conditional independence testing with Von Mises estimator with application to causal discovery (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.