Research map / Large language models / Instruction and fine-tuning
QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces QCircuitBench, a 120k-datapoint benchmark for LLM-driven quantum algorithm design and oracle construction in Qiskit/OpenQASM 3.0 (and Cirq), with automatic syntax and semantic verification, and benchmarks GPT-4o, Llama3, Qwen2.5, DeepSeek-R1 plus a small LoRA fine-tune of Llama3-8B.
Problem
LLMs lack a dataset for designing and implementing quantum algorithms as verifiable quantum circuit code, since existing quantum circuit benchmarks are for NISQ hardware or software tools rather than AI training and evaluation.
Approach
Formulate quantum algorithm design as quantum code generation: each data point pairs a natural-language problem with OpenQASM 3.0 circuits, black-box oracle definitions, Python post-processing, and automated syntax/semantic test-case verification, across 25 algorithms and three task suites.
Key results
- GPT-4o 5-shot achieves an average QASM syntax score of 0.5775 versus 0.2734 at 1-shot on quantum algorithm design tasks.
- Llama3-8B 5-shot reaches average semantic score 0.0323 and GPT-4o 5-shot reaches 0.2006, versus a human baseline average of 0.1374.
- Fine-tuned Llama3 improved most on Grover's algorithm with verification score up by 0.1900, but performance decreased on random circuit tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 5-shot instead of 1-shot prompting | GPT-4o · Bernstein-Vazirani | QASM syntax score: 0.0000 -> 1.0000 | Helped |
| Use 5-shot instead of 1-shot prompting | Qwen 2.5 · Deutsch-Jozsa | Semantic score: 0.0000 -> 0.2854 | Helped |
| Use 5-shot instead of 1-shot prompting | GPT-4o · Quantum algorithm design | Average QASM syntax score: 0.2734 -> 0.5775 | Helped |
| Use 5-shot instead of 1-shot prompting | GPT-4o · Quantum algorithm design | Average semantic score: 0.0245 -> 0.2006 | Helped |
| Use 5-shot instead of 1-shot prompting | Qwen 2.5 · Quantum algorithm design | Average QASM syntax score: 0.2844 -> 0.4307 | Helped |
| Use 5-shot instead of 1-shot prompting | Qwen 2.5 · Quantum algorithm design | Average semantic score: 0.0000 -> 0.0998 | Helped |
| Use 5-shot instead of 1-shot prompting | Llama3 · Quantum algorithm design | Average QASM syntax score: 0.2435 -> 0.4602 | Helped |
| Use 5-shot instead of 1-shot prompting | Llama3 · Quantum algorithm design | Average semantic score: 0.0110 -> 0.0323 | Helped |
| Use 5-shot instead of 1-shot prompting | DeepSeek-R1 · Quantum algorithm design | Average QASM syntax score: 0.0576 -> 0.1768 | Helped |
| Use 5-shot instead of 1-shot prompting | DeepSeek-R1 · Quantum algorithm design | Average semantic score: 0.0126 -> 0.0102 | Hurt |
| Fine-tune on QCircuitBench oracle construction data | Llama3-8B · Grover oracle construction | Aggregated verification score: +0.1900 | Helped |
| Fine-tune on QCircuitBench random circuit synthesis data | Llama3-8B · Clifford and universal random circuits | Aggregated verification score | Hurt |
Surprising findings
- Fine-tuning does not always outperform few-shot learning; performance decreased on Clifford and universal random circuit synthesis.
- DeepSeek-R1 often underperforms due to long chain-of-thought exceeding the context window.
Limitations
- Semantic scores remain very low for most algorithms, especially VQE, QAOA, QFT, Shor and generalized Simon.
- Fine-tuning is only a preliminary demonstration on Llama3-8B for oracle construction and random circuits.
- Quantum information protocols (teleportation, superdense coding, BB84) were not experimentally evaluated.
Experimental setup
Datasets: QCircuitBench
Benchmarks: QCircuitBench
Metrics: BLEU byte perplexity QASM syntax score Python syntax score semantic score gate count ratio shot count ratio time count ratio
Baselines: GPT-4o Llama3-8B Qwen 2.5 DeepSeek-R1 Human study
Builds on: QLoRA LoRA QASMBench MQTBench
Closest papers by meaning
- TensorRL-QAS: Reinforcement learning with tensor networks for improved quantum architecture search (NeurIPS 2025 · similarity 0.89)
- Training-Free Quantum Architecture Search (AAAI 2024 · similarity 0.87)
- AI-Powered Algorithm-Centric Quantum Processor Topology Design (AAAI 2025 · similarity 0.87)
- Rethink the Role of Deep Learning towards Large-scale Quantum Systems (ICML 2025 · similarity 0.87)
- Curriculum reinforcement learning for quantum architecture search under hardware errors (ICLR 2024 · similarity 0.86)
- Benchmarking Quantum Reinforcement Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.