Research map / Large language models / Agents, code and math
AICrypto: Evaluating Cryptography Capabilities of Large Language Models
Yu Wang, Yijian Liu, Liheng Ji, Han Luo, Wenjie Li, Xiaofei Zhou, Chiyun Feng, Puji Wang, Yuhan Cao, Geyuan Zhang, Xiaojian Li, Rongwu Xu and 2 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use frontier reasoning models instead of human experts on multiple-choice crypto questions | o3 · AICrypto MCQ | accuracy (%): 94.1% -> 97.8 | Improved |
| Compare top LLM against human experts on CTF challenges | gemini-2.5-pro · AICrypto CTF | success rate pass@3 (%): 81.2% -> 56.0% | Hurt |
| Compare top LLM against human experts on proof problems | gemini-2.5-pro · AICrypto proofs | scoring rate (%): 94.0% -> 85.4% | Hurt |
| Increase reasoning effort (o3-high vs o3) (ablation) | OpenAI o3 · AICrypto CTF | success rate pass@3 (%): 49.3 -> 54.0 | Improved |
| Use thinking models instead of non-thinking counterparts (ablation) | Claude and Doubao models · AICrypto composite | composite score (0-300) | Improved |
| Use challenges needing advanced math (lattice) instead of known vulnerabilities (RSA) | gemini-2.5-pro · AICrypto CTF categories | success rate pass@3 (%): 78.8 -> 25.0 | Hurt |
| Use two grader models with 3 samples each for automatic proof grading | proof grading · 306 graded samples | Pearson correlation with human: 0.9025 | Improved |
| Let agent run code in a multi-turn environment for CTF | LLM CTF agent · AICrypto CTF | success rate | No clear effect |
Closest papers by meaning
- All Code, No Thought: Language Models Struggle to Reason in Ciphered Language (ICLR 2026 · similarity 0.88)
- Encryption-Friendly LLM Architecture (ICLR 2025 · similarity 0.87)
- TensorCommitments: A Lightweight Verifiable Inference for Large Language Models (NeurIPS 2026 · similarity 0.86)
- DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation (AAAI 2025 · similarity 0.86)
- EncryptedLLM: Privacy-Preserving Large Language Model Inference via GPU-Accelerated Fully Homomorphic Encryption (ICML 2025 · similarity 0.85)
- Benchmarking Large Language Models with Integer Sequence Generation Tasks (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
