Research map / Learning theory and algorithms / Prediction and decision losses
Neuro-Visualizer: A Novel Auto-Encoder-Based Loss Landscape Visualization Method With an Application in Knowledge-Guided Machine Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Neuro-Visualizer, an auto-encoder that learns a non-linear 2-D manifold through training trajectory models for loss landscape visualization, applied to CoPhy-PGNN and PINN case studies in knowledge-guided ML.
Problem
Linear loss-landscape visualizations (e.g., PCA) of training trajectories have low fidelity away from the final model and lack flexibility in scaling and shape, while other non-linear methods lack inverse transforms needed to compute loss on the grid.
Approach
Train an auto-encoder whose 2-D latent grid is decoded back to parameter space, so the loss can be evaluated at any grid point, with optional constraints (location anchoring, grid scaling loss) added as weighted loss terms to shape the manifold.
Key results
- Neuro-Visualizer achieves average relative loss error 0.0095 and projection error 0.0005 on CoPhy-PGNN, versus 1.6782 / 0.2832 for PCA, 4.7250 / 0.0865 for Kernel-PCA, and 0.4295 / 0.2307 for UMAP.
- Neuro-Visualizer visually corroborates Wang et al. (2022) that NTK-based adaptive loss weighting improves boundary-condition loss convergence in PINNs.
- Neuro-Visualizer reveals that LRannealing's trajectory is far from a minimum w.r.t. test loss on the convection PDE with beta=10, contrary to prior claims.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Neuro-Visualizer (auto-encoder manifold) instead of linear/non-linear baselines for landscape visualization | CoPhy-PGNN trajectory landscape · CoPhy-PGNN loss landscape | average relative physics loss error: 0.0095 vs 1.6782 (PCA), 4.7250 (Kernel-PCA), 0.4295 (UMAP) | Helped |
| Use Neuro-Visualizer instead of PCA/Kernel-PCA/UMAP for projection | CoPhy-PGNN trajectory landscape · CoPhy-PGNN loss landscape | average projection error: 0.0005 vs 0.2832 (PCA), 0.0865 (Kernel-PCA), 0.2307 (UMAP) | Helped |
Surprising findings
- Learning rate annealing, which succeeded on Helmholtz and Klein-Gordon in prior work, does not reach a minimum on the convection PDE test loss in the visualized trajectory.
- PCA shows the black-box network's trajectory crossing a contour twice, contradicting its monotonically decreasing loss colors.
Limitations
- Does not scale to large parameter spaces; PINNs with about 10,000 parameters are already relatively small, and computer-vision-scale models are prohibitive.
- Interpolation behavior within the learned auto-encoder manifold is not well understood, so no assumptions are made about it.
- Assessing correctness of loss landscape visualizations is non-trivial and evaluated mainly via reconstruction error and qualitative inspection.
Experimental setup
Datasets: Schrodinger equation trajectories (CoPhy-PGNN) convection PDE PINN training trajectories
Benchmarks: Schrodinger equation (CoPhy-PGNN) convection PDE (PINN)
Metrics: average relative physics loss error average projection error
Baselines: PCA Kernel-PCA UMAP Equal Weights Constant Weights Dynamic Weight Averaging GradNorm Random Loss Weighting Learning Rate Annealing
Builds on: PCA loss landscape visualization (Li et al., 2018) physics-informed neural networks CoPhy-PGNN
Closest papers by meaning
- Visualizing the Loss Landscape of Neural Nets (NeurIPS 2018 · similarity 0.88)
- Taxonomizing local versus global structure in neural network loss landscapes (NeurIPS 2021 · similarity 0.87)
- Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime (ICML 2025 · similarity 0.85)
- Don't trust your eyes: on the (un)reliability of feature visualizations (ICML 2024 · similarity 0.83)
- Learning to cluster neuronal function (NeurIPS 2025 · similarity 0.83)
- Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.