Research map / Large language models / Agents, code and math
STL: Still Tricky Logic (for System Validation, Even When Showing Your Work)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tests whether active learning (with or without runtime-verification feedback) improves humans' ability to validate STL robot policies in a grid-world game, finding ~65% accuracy and no significant condition differences.
Problem
Humans struggle to validate robot policies specified in formal logic (STL), undermining the claim that formal specifications are human-interpretable; the paper asks whether active learning improves validation accuracy.
Approach
Adapts a pedagogical active learning practice to specification validation: participants must generate example trajectories satisfying an STL spec before judging whether it always yields a win, with or without runtime-verification feedback on those trajectories.
Key results
- Overall validation accuracy was 65% ± 15% across 47 analyzed participants (55 recruited, 8 excluded for giving up).
- AL-NF reached 66% ± 16%, AL-WF 67% ± 16%, and control 62% ± 13%; all were significantly above chance but not significantly different from each other (F(2,42)=0.0804, p=0.453).
- Confidence did not differ across conditions (F(2,42)=0.361, p=0.699), and active learning extended engagement time (about 36-40 min vs 20 min for control).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Require trajectory generation without feedback instead of no trajectory generation | ManeuverGame validation task · ManeuverGame validation task | accuracy: 62 ± 13% -> 66 ± 16% | No clear effect |
| Require trajectory generation with runtime-verification feedback instead of no trajectory generation | ManeuverGame validation task · ManeuverGame validation task | accuracy: 62 ± 13% -> 67 ± 16% | No clear effect |
| Add runtime-verification feedback to active learning without feedback | ManeuverGame validation task · ManeuverGame validation task | accuracy: 66 ± 16% -> 67 ± 16% | No clear effect |
| Require trajectory generation without feedback instead of no trajectory generation | ManeuverGame validation task · ManeuverGame validation task | time to completion (minutes): 19.92 ± 9.07 -> 36.31 ± 10.85 | Helped |
| Require trajectory generation with runtime-verification feedback instead of no trajectory generation | ManeuverGame validation task · ManeuverGame validation task | time to completion (minutes): 19.92 ± 9.07 -> 40.37 ± 10.68 | Helped |
Surprising findings
- Active learning with runtime-verification feedback did not improve accuracy over no-feedback or control despite longer engagement.
- Trajectory generation quality was related to correctness differently depending on ground-truth validity of the specification.
Limitations
- Single-session study only; active learning effects in education often require longer periods.
- Abstract grid-world setting with objective game-winning intent rather than real-world stakeholder intent.
- Participant pool mostly formal-methods novices with few domain experts.
Experimental setup
Datasets: ManeuverGame grid-world specification-validation questions
Benchmarks: ManeuverGame validation task
Metrics: validation accuracy confidence calibration Cohen's d time to completion
Baselines: control condition (no active learning)
Builds on: signal temporal logic Bloom's taxonomy Bonwell and Eison active learning
Closest papers by meaning
- Zero-Shot Trajectory Planning for Signal Temporal Logic Tasks (NeurIPS 2025 · similarity 0.87)
- Runtime Monitoring of Perception-Based Autonomous Systems via Embedding Temporal Logic (NeurIPS 2026 · similarity 0.86)
- TeLoGraF: Temporal Logic Planning via Graph-encoded Flow Matching (ICML 2025 · similarity 0.85)
- TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks (NeurIPS 2026 · similarity 0.85)
- Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learning (ICML 2024 · similarity 0.85)
- Automaton Constrained Q-Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.