Research map / Label-efficient and robust learning / Recognition and long-tail
SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SNIP, a dual-encoder contrastive pre-training model aligning symbolic equations with their numeric data, then applies it to cross-modal property prediction and symbolic regression via a decoder and latent space optimization.
Problem
Existing models handle either symbolic equations or numeric data, and are trained supervised for specific tasks; there is no task-agnostic pre-training that jointly understands symbolic expressions and their numeric observations.
Approach
Dual Transformer encoders (symbolic and numeric) trained with a symmetric CLIP-style InfoNCE loss on about 60M synthetic (equation, datapoints) pairs; the pretrained numeric encoder is then reused for symbolic regression with an expression decoder and latent space optimization.
Key results
- On property prediction with 10K training equations, SNIP (finetuned) reduces NCR NMSE to 0.0683 versus 0.5299 for the supervised model (Acc_0.1 92.1% vs 56.5%).
- On SRBench Feynman datasets, SNIP achieves accuracy 0.882 vs 0.798 for AIFeynman with complexity 31.63 vs 69.87 for Operon.
- On Strogatz datasets SNIP reaches top-tier accuracy of 0.928, outperforming leading baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pretrained SNIP representations instead of random-initialized supervised training | Cross-modal property prediction · Non-Convexity Ratio | NMSE: 0.5299 -> 0.0731 | Helped |
| Use pretrained SNIP representations instead of random-initialized supervised training | Cross-modal property prediction · Upwardness | NMSE: 0.5356 -> 0.0540 | Helped |
| Use pretrained SNIP representations instead of random-initialized supervised training | Cross-modal property prediction · Normalized Average y | NMSE: 1.0406 -> 0.4532 | Helped |
| Use pretrained SNIP representations instead of random-initialized supervised training | Cross-modal property prediction · Log Oscillations | NMSE: 0.3079 -> 0.0683 | Helped |
| Fine-tune the pretrained SNIP encoder (ablation) | SNIP for cross-modal property prediction · Non-Convexity Ratio | NMSE: 0.0731 -> 0.0683 | Helped |
| Fine-tune the pretrained SNIP encoder (ablation) | SNIP for cross-modal property prediction · Upwardness | NMSE: 0.0540 -> 0.0400 | Helped |
| Fine-tune the pretrained SNIP encoder (ablation) | SNIP for cross-modal property prediction · Log Oscillations | Acc_0.1: 92.6% -> 92.6% | No clear effect |
| Fine-tune the pretrained SNIP encoder (ablation) | SNIP for cross-modal property prediction · Normalized Average y | NMSE: 0.4532 -> 0.4074 | Helped |
| Use SNIP rather than the supervised model with 100 training samples | NCR prediction · Low-data regime | R^2: 0.292; above 0.745 | Helped |
| Apply latent space optimization after training the SNIP-based equation generator (ablation) | SNIP symbolic regression · SRBench | R^2 | Helped |
| Use SNIP for symbolic regression | Symbolic regression · ODE-Strogatz | R^2: 0.928 | Helped |
| Use SNIP rather than Operon | Symbolic regression · Black-box | Model complexity (nodes): 64.95 -> 47.52 | Helped |
Surprising findings
- The supervised baseline's Normalized Average y NMSE (1.0406) is worse than chance level (1.0), indicating it fails on that property.
Limitations
- Limited to input dimension D<=10 due to exponential expression complexity and the data generation protocol.
- Struggles with data patterns that cannot be expressed as closed-form functions.
- Performance tied to the predefined synthetic data generation protocol and operator vocabulary.
Experimental setup
Datasets: synthetic symbolic-numeric function pairs PMLB datasets via SRBench (Feynman, Strogatz, Black-box)
Benchmarks: SRBench (Feynman, Strogatz, Black-box) cross-modal property prediction (NCR, Upwardness, Normalized Average y, Log Oscillations)
Metrics: NMSE Acc_0.1 R^2 expression complexity (number of nodes)
Baselines: Supervised transformer (same encoder, random init) E2E Operon AIFeynman SRBench methods
Builds on: CLIP ClipCap End-to-end symbolic regression transformer (E2E) SRBench Set transformer style permutation-invariant numeric encoder
Closest papers by meaning
- LLM-SR: Scientific Equation Discovery via Programming with Large Language Models (ICLR 2025 · similarity 0.87)
- A Neural-Guided Dynamic Symbolic Network for Exploring Mathematical Expressions from Data (ICML 2024 · similarity 0.87)
- SMILE: Bridging Continuous Optimization and Discrete Symbolic Recovery (NeurIPS 2026 · similarity 0.86)
- Symbolic Neural Ordinary Differential Equations (AAAI 2025 · similarity 0.85)
- LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models (ICML 2025 · similarity 0.85)
- Ab Initio Nonparametric Variable Selection for Scalable Symbolic Regression with Large $p$ (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.