Research map / Architectures and efficiency / Pruning and efficient tuning
Optimistic Verifiable Training by Controlling Hardware Nondeterminism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an optimistic verifiable training scheme that controls GPU hardware nondeterminism by training in higher precision, rounding intermediate computations, and logging the trainer's rounding decisions with an adaptive threshold so auditors can replicate training exactly.
Problem
Optimistic verifiable training fails because nondeterminism between GPU types makes exact replication of training impossible, so weight-comparison audits are non-robust; proof-based alternatives are too expensive to scale.
Approach
Trainer trains in FP64 (higher than target FP32), rounds after each intermediate computation, and logs rounding directions for outputs far enough from the rounding boundary (adaptive threshold tau); the auditor applies the logged corrections via a rev operation so that weights match bit-for-bit across GPUs, and an interactive Merkle-tree verification game localizes disputes.
Key results
- Exact training replication at FP32 across all trainer-auditor pairs of A40, Titan XP, and RTX 2080 Ti GPUs for ResNet-50 and GPT-2 training/fine-tuning.
- Efficient encoding reduces rounding-log storage by 77%; GPT-2 log is 18MB versus 2.78GB of weights in proof-of-learning (>140x reduction).
- Auditor needs only 20-25 rev corrections per training step at b=32 (2e-6% to 9e-6% of logged values); training time overhead of 1.2-1.4x for trainer and 1.3-1.7x for auditor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use rounding logs and auditor corrections to synchronize intermediate rounding decisions | Optimistic verifiable training · ResNet-50 and GPT-2 across A40, Titan XP, and RTX 2080 Ti | cross-GPU training replication | Helped |
| remove auditor rounding corrections while retaining FP64 training and FP32 rounding (ablation) | Optimistic verifiable training · ResNet-50 and GPT-2 cross-GPU training | cross-GPU training replication | Hurt |
| use standard FP32 training instead of hardware-nondeterminism control (ablation) | ResNet-50 and GPT-2 · cross-GPU training | weight divergence | Hurt |
| use efficient encoding instead of naive log encoding (ablation) | Optimistic verifiable training · ResNet-50 and GPT-2, 1 training step | storage: 456 MB -> 105 MB; 92 MB -> 22 MB | Helped |
| add Zip compression to efficiently encoded rounding logs (ablation) | Optimistic verifiable training · ResNet-50 and GPT-2, 1 training step | storage: 105 MB -> 96 MB; 22 MB -> 20 MB | Helped |
| reduce rounding amount from b=32 to b=26 (ablation) | Optimistic verifiable training · ResNet-50 and GPT-2 | test performance | No clear effect |
| reduce rounding amount from b=32 to b=26 (ablation) | Optimistic verifiable training · ResNet-50 and GPT-2, 1 training step | storage: 96 MB -> 91 MB; 20 MB -> 18 MB | Helped |
| reduce rounding amount from b=32 to b=26 (ablation) | Optimistic verifiable training · ResNet-50 and GPT-2 | training time | Hurt |
| increase rounding aggressiveness to b=27 (ablation) | Optimistic verifiable training · ResNet-50 and GPT-2 | auditor rev corrections: 0 | Helped |
| apply optimistic verifiable training instead of the zero-knowledge proof system | Logistic Regression training · Logistic Regression, one training pass | storage: 350 MB -> 106 KB | Helped |
| apply optimistic verifiable training instead of the zero-knowledge proof-of-training protocol | VGG-11 training · VGG-11, one training iteration | training and auditing time: 22 min. -> 6 sec. | Helped |
| use GPU nondeterminism instead of data ordering as the source of model variation | ResNet-50 · model divergence comparison | l2 weight difference: $133.2 \pm 9$ -> $1.1 \pm 0.07$ | Helped |
Surprising findings
- Severe rounding (b=27) completely eliminated hardware nondeterminism in the tested tasks, requiring no auditor corrections.
- Only a tiny fraction of logged rounding decisions (around 1e-6 to 1e-5) are actually needed by the auditor, indicating large storage slack.
Limitations
- Requires trust in the third-party auditor and cooperation of the trainer; trainer must share weights with auditors, unsuitable for closed-source models.
- Storage cost of rounding logs is large (every intermediate output logged), and most logged decisions are unnecessary.
- Adversarial trainer could select among rounding decisions to steer outcomes within the set of valid models; security under this threat model not fully characterized.
Experimental setup
Datasets: CIFAR-10 Shakespeare corpus
Benchmarks: CIFAR-10 Shakespeare corpus
Metrics: exact weight replication across GPUs l2 weight divergence storage cost (MB/GB) training time (s) number of rev corrections test accuracy perplexity test loss
Baselines: proof-of-learning zero-knowledge logistic regression verifiable training zkML VGG-11 proof-of-training MPC-based GPT-2 secure inference
Builds on: verification game proof-of-learning Merkle tree
Closest papers by meaning
- ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload (NeurIPS 2026 · similarity 0.84)
- TensorCommitments: A Lightweight Verifiable Inference for Large Language Models (NeurIPS 2026 · similarity 0.84)
- Certified but Private: Scalable Zero-Knowledge Proofs for the Formal Verification of Neural Networks (NeurIPS 2026 · similarity 0.83)
- MIBP-Cert: Certified Training against Data Perturbations with Mixed-Integer Bilinear Programs (NeurIPS 2025 · similarity 0.82)
- Training Verification-Friendly Neural Networks via Neuron Behavior Consistency (AAAI 2025 · similarity 0.82)
- Probing RLVR Training Instability through the Lens of Objective-Level Hacking (ICML 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.