Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
On Neural Networks as Infinite Tree-Structured Probabilistic Graphical Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Constructs an infinite-width tree-structured Markov network whose inference exactly reproduces sigmoid DNN forward passes and gradients, and uses this to motivate HMC-based contrastive-divergence fine-tuning for calibration.
Problem
Deep neural networks lack precise probabilistic semantics as probabilistic graphical models; it is unclear what joint distribution a DNN's forward pass approximates, and how PGM inference tools could be applied to DNNs.
Approach
Unroll any sigmoid DNN into an infinite-width treewidth-1 Markov network (copying parents and dividing weights by L); as L goes to infinity, marginal inference equals the sigmoid forward pass and the marginal likelihood gradient equals the cross-entropy gradient. This motivates HMC over continuous hidden variables with a normal approximation for contrastive-divergence fine-tuning.
Key results
- On synthetic BN/MN data, HMC with L=10 fine-tuning gave lower MAE than plain DNN in several settings (e.g., BN (0.3) at 100 epochs: 5.300e-3 vs 6.593e-3), with Gibbs often no better than DNN.
- On Covertype label 1 at 100 epochs, HMC-10 reduced test ECE from 4.207e-2 (DNN) to 2.352e-2.
- Gibbs fine-tuning sometimes performed worse than the DNN, e.g., Covertype label 2 at 100 epochs: 7.796e-2 vs 4.268e-2 for DNN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with HMC using L=10 rather than leave the DNN unfine-tuned | DNN · Synthetic BN (0.3), 100 train epochs | MAE (× 10^-3): 6.593 -> 5.300 | Helped |
| Fine-tune with HMC using L=10 rather than leave the DNN unfine-tuned | DNN · Synthetic BN (0.3), 1000 train epochs | MAE (× 10^-3): 34.44 -> 23.53 | Helped |
| Fine-tune with HMC using L=10 rather than leave the DNN unfine-tuned | DNN · Synthetic BN (3), 100 train epochs | MAE (× 10^-3): 72.76 -> 76.54 | Hurt |
| Fine-tune with HMC using L=10 rather than leave the DNN unfine-tuned | DNN · Synthetic MN (0.3), 100 train epochs | MAE (× 10^-3): 6.031 -> 4.515 | Helped |
| Fine-tune with HMC using L=10 rather than leave the DNN unfine-tuned | DNN · Synthetic MN (1), 100 train epochs | MAE (× 10^-3): 9.671 -> 8.887 | Helped |
| Fine-tune with HMC using L=10 rather than leave the DNN unfine-tuned | DNN · Covertype label 1, 100 train epochs | ECE (× 10^-2): 4.207 -> 2.352 | Helped |
| Fine-tune with HMC using L=10 rather than leave the DNN unfine-tuned | DNN · Covertype label 2, 100 train epochs | ECE (× 10^-2): 4.268 -> 7.719 | Hurt |
| Fine-tune with HMC using L=10 rather than leave the DNN unfine-tuned | DNN · Covertype label 2, 1000 train epochs | ECE (× 10^-2): 6.634 -> 5.233 | Helped |
| Use L=100 rather than L=10 for HMC fine-tuning (ablation) | HMC fine-tuning · Synthetic BN (0.3), 100 train epochs | MAE (× 10^-3): 5.300 -> 6.864 | Hurt |
| Use L=1000 rather than L=10 for HMC fine-tuning (ablation) | HMC fine-tuning · Synthetic BN (0.3), 100 train epochs | MAE (× 10^-3): 5.300 -> 6.658 | Hurt |
| Use L=100 rather than L=10 for HMC fine-tuning (ablation) | HMC fine-tuning · Synthetic MN (0.3), 100 train epochs | MAE (× 10^-3): 4.515 -> 6.382 | Hurt |
| Use L=1000 rather than L=10 for HMC fine-tuning (ablation) | HMC fine-tuning · Synthetic MN (0.3), 100 train epochs | MAE (× 10^-3): 4.515 -> 6.070 | Hurt |
Surprising findings
- Gibbs sampling in the constructed PGM, the theoretically consistent method, often calibrated worse than the plain DNN.
- Larger HMC L (1000) often reverted to DNN-level calibration, with small gains only for L=10.
Limitations
- The infinite tree construction is impractical; the result is an approximation to the direct PGM, and only sigmoid activations are proven exactly.
- Extension to ReLU, normalization layers and other activations is argued informally rather than proven.
- Empirical evaluation is limited to small synthetic datasets and one tabular dataset.
Experimental setup
Datasets: Synthetic BN/MN datasets (1000 points each) Covertype
Benchmarks: Synthetic BN/MN calibration Covertype (labels 1 and 2)
Metrics: mean absolute error expected calibration error t-test p-values
Baselines: SGD training Gibbs sampling HMC with L=100 HMC with L=1000
Builds on: contrastive divergence variable elimination conditional random fields Hamiltonian Monte Carlo
Closest papers by meaning
- PGM-Explainer: Probabilistic Graphical Model Explanations for Graph Neural Networks (NeurIPS 2020 · similarity 0.88)
- On the Expressive Power of Tree-Structured Probabilistic Circuits (NeurIPS 2024 · similarity 0.86)
- Explaining, Fast and Slow: Abstraction and Refinement of Provable Explanations (ICML 2025 · similarity 0.86)
- Sum-Product-Set Networks: Deep Tractable Models for Tree-Structured Graphs (ICLR 2024 · similarity 0.86)
- Probabilistic Neural Circuits (AAAI 2024 · similarity 0.85)
- [Re] GNNInterpreter: A probabilistic generative model-level explanation for Graph Neural Networks (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.