Research map / Large language models / Language, safety and interpretability
jqBench: a benchmark for reading and editing JSON from natural language and/or examples
Gust Verbruggen, Chris Parnin, Vu Le, Sumit Gulwani
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| write the solution in Python instead of jq | Phi 4 · jqStack | value match: 10% -> 54% | Improved |
| write the solution in Python instead of jq | GPT-5 · jqStack | value match: 68% -> 82% | Improved |
| write the solution in Python instead of jq | Opus 4.1 · jqStack | value match: 76% -> 83% | Improved |
| add the run_code tool (ablation) | GPT-5 · jqStack | value match: -53% | Hurt |
| add the run_code tool (ablation) | Opus 4.1 · jqStack | value match: -22% | Hurt |
| add documentation search tools (ablation) | Opus 4.1 · jqStack | value match: 76% -> 31% | Hurt |
| add documentation tools (ablation) | GPT-5 · jqStack | value match: 0.68 -> 0.68 | No clear effect |
| allow more feedback iterations | jq agents · jqStack | cumulative value match | Improved |
| use only input-output examples without natural language | Opus 4.1 jq · jqStack | value match: 0.76 -> 0.71 | Hurt |
| use only input-output examples without natural language | GPT-5 jq · jqStack | value match: 0.68 -> 0.63 | Hurt |
| add implicit feedback to Opus 4.1 jq | Opus 4.1 · jqSpider | value match: 0.77 -> 0.81 | Improved |
| add implicit feedback to GPT-5 jq | GPT-5 · jqSpider | value match: 0.72 -> 0.78 | Improved |
Closest papers by meaning
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.84)
- PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation (NeurIPS 2025 · similarity 0.84)
- Structure-Guided Large Language Models for Text-to-SQL Generation (ICML 2025 · similarity 0.83)
- LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning (AAAI 2026 · similarity 0.83)
- The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution (ICLR 2026 · similarity 0.83)
- Earley-Driven Dynamic Pruning for Efficient Structured Decoding (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
