Research map / Large language models / Reasoning and chain of thought
Auditor-Assisted Summary-Channel Verification for Hosted LLM Identity Substitution
Ziyi Zhang, Ziyao Wang, Guoheng Sun, Ang Li, Jian Li
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an auditor-assisted regime for verifying whether a hosted LLM service serves the claimed model identity, using pre-enrolled verification material and audit-time checks on released outputs only.
Problem
Hosted LLM providers may claim a model identity that differs from the configuration actually serving requests, and external users and auditors cannot inspect the served model at runtime.
Approach
A trusted auditor enrolls claimed identities and prepares verification material before deployment; audit-time verification then uses only the claimed identity and the released output, with no runtime access to the service.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use identity-specific keyed carrier sets with adapters instead of generic watermarks (Unigram, Gumbel-Max, SynthID) (ablation) | SumMark · Math substitution detection, 7 pairs | TPR@1% FPR: Unigram 0.4730 -> 1.0000 on Qwen-7B to Qwen-3B | Improved |
| summary-channel watermark instead of unkeyed text classifiers | SumMark · General substitution detection | average TPR@1% FPR: 0.5439 -> 0.9750 (vs BERT) | Improved |
| summary watermarking on task-facing answers (ablation) | SumMark · Math and Code | answer accuracy / Pass@1: Qwen-7B Math 0.5034 -> 0.5062; Code 0.1920 -> 0.1830 | No clear effect |
| summary watermarking on General summaries (ablation) | SumMark · General | perplexity: Qwen-1.5B 29.3089 -> 25.6346 | Improved |
| bounded edits (shuffle, deletion, truncation) to released summary (ablation) | SumMark · Math robustness | TPR@1% FPR: 0.9994 -> 0.9992 | No clear effect |
| bilingual rewriting of summary (ablation) | SumMark · Math robustness | TPR@1% FPR: 0.9994 -> 0.7900 (worst 0.5740) | Hurt |
| adapter distillation spoofing attack (ablation) | SumMark · Math robustness | ASR@1% FPR: average 0.1651, worst 0.6290 | Hurt |
| watermark stealing spoofing attack (ablation) | SumMark · Math robustness | ASR@1% FPR: average 0.0417 | No clear effect |
| quantize served model from FP16 to INT8 (small model) (ablation) | SumMark precision diagnostic · Qwen-1.5B Math | TPR@1% FPR: INT8 0.1132 vs INT4 0.3417 | Hurt |
| more aggressive quantization (INT4 vs INT8) (ablation) | SumMark precision diagnostic · Qwen-1.5B, Math | TPR@1% FPR: 0.1132 -> 0.3417 | Improved |
| use SumMark instead of T5 text classifier on templated Code summaries | SumMark attribution · Code attribution | accuracy: T5 0.9841 vs SumMark 0.9678 | Hurt |
Closest papers by meaning
- Portcullis: A Scalable and Verifiable Privacy Gateway for Third-Party LLM Inference (AAAI 2025 · similarity 0.84)
- When to Trust the Cheap Check: Weak and Strong Verification for Reasoning (ICML 2026 · similarity 0.84)
- The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text (ICML 2025 · similarity 0.83)
- PrivAuditor: Benchmarking Data Protection Vulnerabilities in LLM Adaptation Techniques (NeurIPS 2024 · similarity 0.83)
- Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain (ICLR 2024 · similarity 0.83)
- Combating Data Laundering in LLM Training (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
