Research map / Large language models / Agents, code and math
Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines self-consistency for Code LLMs (NL-to-PL and PL-to-NL round trips) and proposes IdentityChain, a greedy-decoding framework with a Test Output Match score to evaluate self-consistency alongside accuracy across eleven code LLMs.
Problem
Conventional evaluations of Code LLMs measure NL-to-PL and PL-to-NL accuracy in isolation and ignore whether a model stays consistent when translating its own outputs back and forth between natural language and code.
Approach
Formalizes self-consistency as preserving semantics along an identity chain of model-generated NL-to-PL and PL-to-NL translations, approximates semantic equality with a test-output-match (TOM) metric, and uses greedy decoding with early stopping to make evaluation efficient.
Key results
- Across eleven Code LLMs, strong self-consistency within 5 steps (SSC@5) drops by up to 78.0% relative to Pass@1 (e.g., StarChat-Beta on HumanEvalPlus).
- GPT-4 reaches SC@5 of 76.1 on HumanEvalPlus, about 51.2% higher than GPT-3.5 despite only a 4.2% relative gain in Pass@1 (74.8 vs 71.8).
- TOM correlates better with human judgment than EM, CodeBLEU, P/FM, BLEU, ROUGE-L, chrF and BERTScore (Pearson 0.461 for NL-to-PL and 0.500 for PL-to-NL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate self-consistency over five identity-chain steps rather than one step | Code LLMs evaluated with IdentityChain · HumanEvalPlus and MBPP Sanitized | self-consistency (SCon): up to 43.8% | Hurt |
| Evaluate strong self-consistency over five identity-chain steps rather than initial accuracy | Code LLMs evaluated with IdentityChain · HumanEvalPlus and MBPP Sanitized | strong self-consistency (SSCon): up to 78.0% | Hurt |
| Use TOM instead of Exact Match for estimating semantic equivalence | IdentityChain self-consistency evaluation · GPT-3.5 one-step chains | Pearson correlation with human judgment: .285 -> .461 | Helped |
| Use TOM instead of CodeBLEU for estimating semantic equivalence | IdentityChain self-consistency evaluation · GPT-3.5 one-step chains | Pearson correlation with human judgment: .179 -> .461 | Helped |
| Use TOM instead of Pass/Fail Match for estimating semantic equivalence | IdentityChain self-consistency evaluation · GPT-3.5 one-step chains | Pearson correlation with human judgment: .294 -> .461 | Helped |
| Use TOM instead of BLEU for PL-to-NL evaluation | IdentityChain · GPT-3.5 one-step chains; 117 problems | Pearson correlation with human judgment: .285 -> .500 | Helped |
| Use TOM instead of ROUGE-L for PL-to-NL evaluation | IdentityChain · GPT-3.5 one-step chains; 117 problems | Pearson correlation with human judgment: .254 -> .500 | Helped |
| Use TOM instead of chrF for PL-to-NL evaluation | IdentityChain · GPT-3.5 one-step chains; 117 problems | Pearson correlation with human judgment: .271 -> .500 | Helped |
| Use TOM instead of BERTScore for PL-to-NL evaluation | IdentityChain · GPT-3.5 one-step chains; 117 problems | Pearson correlation with human judgment: .381 -> .500 | Helped |
| Increase model size from StarCoderBase-7B to StarCoderBase-15B | StarCoderBase · HumanEvalPlus and MBPP Sanitized | strong self-consistency (SSCon@5): 8.6 -> 8.0; 23.0 -> 21.0 | Hurt |
| Increase model size from StarCoderBase-7B to StarCoderBase-15B | StarCoderBase · HumanEvalPlus and MBPP Sanitized | Pass@1: 24.5 -> 27.0; 43.6 -> 44.0 | Helped |
| Replace greedy decoding with higher-temperature decoding | Code LLMs evaluated with IdentityChain · HumanEvalPlus and MBPP Sanitized | SCon@5 | Hurt |
Surprising findings
- More accurate models are not necessarily more self-consistent; StarChat-Beta (15B) has higher Pass@1 than CodeLlama-Instruct-7B but lower SC@5.
- Most self-consistency failures appear within the first three steps of the chain.
Limitations
- Self-consistency is only evaluated for Python and English, limiting generality to other languages.
- Greedy decoding is used with closed-source models at temperature 0, which is not truly deterministic; results at other temperatures only partially verified.
- Relies on test cases for TOM, so approximation of semantic equality is imperfect; human judgment used on a single small sample (163 problems).
Experimental setup
Datasets: HumanEvalPlus-Mini-v0.1.6 MBPP sanitized test split
Benchmarks: HumanEvalPlus MBPP Sanitized
Metrics: Pass@1 self-consistency (SC) score strong self-consistency (SSC) score TOM score Pearson correlation Spearman correlation Kendall-Tau correlation
Baselines: Exact Match CodeBLEU Pass/Fail Match BLEU ROUGE-L chrF BERTScore
Builds on: HumanEval MBPP CodeBLEU Pass@K
Closest papers by meaning
- CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules (ICLR 2024 · similarity 0.89)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.88)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.88)
- BaxBench: Can LLMs Generate Correct and Secure Backends? (ICML 2025 · similarity 0.88)
- VERSE: Verification-based Self-Play for Code Instructions (AAAI 2025 · similarity 0.88)
- Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.