Research map / Large language models / Reasoning and chain of thought
TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TOPLOC, a locality-sensitive hashing scheme that encodes top-k activation indices and values as a modular polynomial congruence, enabling compact, hardware-robust verification of LLM inference.
Problem
Users of open-weights LLMs served by inference providers cannot verify that the claimed model, prompt, and precision were used; cryptographic proofs are too expensive and storing raw activations is storage-prohibitive.
Approach
Hash the top-k largest-magnitude hidden-state activations by interpolating a polynomial congruence over indices reduced modulo an injective modulus, storing only k two-byte coefficients per 32 generated tokens; validation compares exponent and mantissa statistics of recomputed top-k values, exploiting that large-magnitude values are robust to GPU nondeterminism and reordering.
Key results
- Same-model, same-precision validation passes with worst-case exponent mismatch below threshold 38, while different models, prompt alterations, and precisions all fail with best-case mismatch above threshold, giving 100% classification in reported experiments.
- Proof storage is 256-258 bytes per 32 new tokens (about 8 bytes per token), a roughly 1000x reduction versus storing bf16 final hidden states (8192 bytes per token) for Llama-3.1-8B.
- Validation takes about 81 ms versus zkLLM's 803 s validation and 986 s commitment for Llama-2-13B, with 0% false positive and negative rates in the tested settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TopLoc polynomial encoding instead of storing raw activations | LLM inference proofs · Llama-3.1-8B-Instruct | proof storage: 8192 bytes per token -> 8 bytes per token; $1024\times$ reduction | Helped |
| Use TopLoc rather than zkLLM | LLM inference verification · Llama-2-13B inference | committing overhead per token: $986s$ -> $0.26ms$ | Helped |
| Use TopLoc rather than zkLLM | LLM inference verification · Llama-2-13B inference | commitment size per token: 11 MB -> 8B | Helped |
| Validate across different GPU and tensor-parallel configurations | TopLoc · Llama-3.1-8B-Instruct, INTELLECT-1-Instruct, and Gemma-2-9b-it | positive-sample classification | No clear effect |
| Validate with different attention kernel implementations | TopLoc · Flash Attention 2, SDPA, and FlexAttention | positive-sample classification | No clear effect |
| Use a different model for validation than for generation | TopLoc · Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct, INTELLECT-1-Instruct, and Gemma-2-9b-it | model mismatch detection: Exponent mismatch minimums include $125\downarrow\;(97.66\%)$ and $128\downarrow\;(100\%)$ | Helped |
| Alter the system prompt | TopLoc · Llama-3.1-8B-Instruct prompt alteration tests | prompt alteration detection: Minimum exponent mismatches: $95\;(74.22\%)$, $74\;(57.81\%)$, and $67\;(52.34\%)$ | Helped |
| Generate with bf16 and validate with fp32 | TopLoc · Llama-3.1-8B-Instruct precision mismatch | precision mismatch detection: Minimum mantissa differences: $27892.02\downarrow$ and $21683\downarrow$ | Helped |
| Generate with fp32 and validate with bf16 | TopLoc · Llama-3.1-8B-Instruct precision mismatch | precision mismatch detection: Maximum exponent mismatch $14\uparrow\;(10.94\%)$ | No clear effect |
Surprising findings
- Fp32 generations validated with bf16 pass, while bf16 generations always fail when validated with fp32.
Limitations
- Cannot detect speculative decoding or KV-cache compression type; fp8 vs bf16 margin is small.
- Subtle modifications such as minor prompt tweaks or slight fine-tuning are harder to detect.
- Adversarial prompt mining could exploit unstable generations to induce validation failures.
Experimental setup
Datasets: UltraChat
Metrics: top-k mismatch exponent mismatch mantissa difference mean and median false positive rate false negative rate storage per token validation time
Baselines: zkLLM SVIP Raw Activations Verisplit
Builds on: zkLLM SVIP Verisplit
Closest papers by meaning
- TensorCommitments: A Lightweight Verifiable Inference for Large Language Models (NeurIPS 2026 · similarity 0.90)
- VIA-SD: Verification via Intra-Model Routing for Speculative Decoding (ICML 2026 · similarity 0.85)
- Hidden No More: Attacking and Defending Private Third-Party LLM Inference (ICML 2025 · similarity 0.85)
- MPCache: MPC-Friendly KV Cache Eviction for Efficient Private LLM Inference (NeurIPS 2025 · similarity 0.85)
- Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference (ICML 2025 · similarity 0.84)
- EncryptedLLM: Privacy-Preserving Large Language Model Inference via GPU-Accelerated Fully Homomorphic Encryption (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.