Research map / Large language models / Agents, code and math
RvLLM: LLM Runtime Verification with Domain Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RvLLM, a runtime verification framework with a domain-specific specification language (ESL) that checks LLM outputs against expert rules via LLM-based interpretation and forward chaining, evaluated on legal violation detection, numerical comparison, and inequality solving.
Problem
LLMs produce inconsistent or erroneous outputs in high-stakes domains, and existing LLM testing and verification approaches are generic and do not incorporate domain-specific constraints.
Approach
A simple specification language (ESL) lets domain experts write rule-like predicate-logic constraints with natural language predicate descriptions; a perception LLM instantiates propositions from context and outputs, forward chaining over normalized rules infers new facts, and follow-up queries to the target LLM check consistency.
Key results
- Integrating RvLLM raises violation-detection TPR by 15.7% to 50.2% across 13 LLMs (e.g., DeepSeek-V3 78.3% to 94.0%).
- On 100 numerical comparison questions, RvLLM detects nearly all erroneous outputs; e.g., GPT 4.1 mini and Gemini 2.0 Flash reach 88.9% and 100% TPR with 100% TNR.
- On inequality solving, RvLLM achieves up to 50% TPR for detecting erroneous solutions using DeepSeek-V3 as perception agent, with only three specifications.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RvLLM to the standalone target LLM | LLMs on Singapore Rapid Transit Systems Act violation detection · Singapore Rapid Transit Systems Act violation detection | TPR: +15.7% to +50.2% | Helped |
| Add RvLLM to Qwen 2.5 (7B) | Qwen 2.5 (7B) · Singapore Rapid Transit Systems Act violation detection | TPR: 16.0% -> 61.2% | Helped |
| Add RvLLM to DeepSeek-V3 | DeepSeek-V3 · Singapore Rapid Transit Systems Act violation detection | TNR: 87.0% -> 82.6% | Hurt |
| Use RvLLM rather than SelfCheckGPT | Qwen plus · Numerical comparison | TNR: 96.7% -> 100% | Helped |
| Use RvLLM rather than LLM-as-a-judge | Qwen plus · Numerical comparison | TNR: 0 -> 100% | Helped |
| Use DeepSeek-V3 instead of Qwen 2.5 (32B) as the perception agent | RvLLM · Inequality solving (Qwen 2.5 (32B) target) | TPR: 25% -> 50% | Helped |
| Use DeepSeek-V3 instead of Qwen 2.5 (32B) as the perception agent | RvLLM · Inequality solving (Qwen 2.5 (7B) target) | TPR: 20% -> 48% | Helped |
| Use DeepSeek-V3 instead of Qwen 2.5 (32B) as the perception agent | RvLLM · Inequality solving across evaluated models | TPR: up to 50% | Helped |
| Use three expert-defined specifications for inequality verification | RvLLM · Inequality solving | TPR: 50% | Mixed |
Surprising findings
- Integrating RvLLM can lower TNR for some models (e.g., Qwen 2.5 7B and GPT 4.1), attributed to perception-agent errors.
- Qwen 2.5 (7B) failed all interpretation abstraction in numerical comparison tasks.
Limitations
- Limited expressiveness of ESL restricts encoding of complex domain knowledge.
- Soundness depends on faithful expert encoding of domain knowledge.
- Perception-agent errors cause false positives and reduced TNR; inequality specifications cover only three error types.
Experimental setup
Datasets: Singapore Rapid Transit Systems Act scenarios (304 cases) random numerical comparison pairs (100) A-Level H2 Mathematics inequality problems (40)
Benchmarks: Singapore Rapid Transit Systems Act violation detection numerical comparison inequality solving
Metrics: TPR TNR error detection count
Baselines: SelfCheckGPT LLM-as-a-judge
Builds on: SelfCheckGPT forward chaining unit propagation
Closest papers by meaning
- Rethinking Verification for LLM Code Generation: From Generation to Testing (NeurIPS 2025 · similarity 0.88)
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.88)
- The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements (NeurIPS 2025 · similarity 0.88)
- SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types (NeurIPS 2024 · similarity 0.88)
- Grammars of Formal Uncertainty: When to Trust LLMs in Automated Reasoning Tasks (NeurIPS 2025 · similarity 0.87)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.