AI Paper Atlas

Research map / Large language models / Agents, code and math

AICrypto: Evaluating Cryptography Capabilities of Large Language Models

Yu Wang, Yijian Liu, Liheng Ji, Han Luo, Wenjie Li, Xiaofei Zhou, Chiyun Feng, Puji Wang, Yuhan Cao, Geyuan Zhang, Xiaojian Li, Rongwu Xu and 2 more

ICML 2026 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use frontier reasoning models instead of human experts on multiple-choice crypto questionso3 · AICrypto MCQaccuracy (%): 94.1% -> 97.8Improved
Compare top LLM against human experts on CTF challengesgemini-2.5-pro · AICrypto CTFsuccess rate pass@3 (%): 81.2% -> 56.0%Hurt
Compare top LLM against human experts on proof problemsgemini-2.5-pro · AICrypto proofsscoring rate (%): 94.0% -> 85.4%Hurt
Increase reasoning effort (o3-high vs o3) (ablation)OpenAI o3 · AICrypto CTFsuccess rate pass@3 (%): 49.3 -> 54.0Improved
Use thinking models instead of non-thinking counterparts (ablation)Claude and Doubao models · AICrypto compositecomposite score (0-300)Improved
Use challenges needing advanced math (lattice) instead of known vulnerabilities (RSA)gemini-2.5-pro · AICrypto CTF categoriessuccess rate pass@3 (%): 78.8 -> 25.0Hurt
Use two grader models with 3 samples each for automatic proof gradingproof grading · 306 graded samplesPearson correlation with human: 0.9025Improved
Let agent run code in a multi-turn environment for CTFLLM CTF agent · AICrypto CTFsuccess rateNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy