Research map / Large language models / Agents, code and math
TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TypyBench, a benchmark of 50 Python repositories with two metrics (TypeSim, a semantic type-similarity score, and TypeCheck, mypy-error-based consistency) to evaluate LLM type inference at repository level.
Problem
Existing type inference evaluation relies on exact or parametric-outer-type matching and judges types in isolation, missing semantic similarity and repository-wide type consistency of LLM-predicted Python annotations.
Approach
Score predicted type stubs with a continuous Jaccard-over-method-sets similarity computed recursively over generic type trees and optimal matching for unions, and measure consistency as the number of mypy errors the predicted types induce across the repository.
Key results
- Top API models reach TypeSim around 0.80 (GPT-4o 0.804) but show large mypy error counts; Claude achieves the lowest TypeCheck score (127.1 average vs. ground truth 141.8).
- Exact match drops from 0.699 at depth 1 to 0.211 at depth 5, while TypeSim falls only from 0.725 to 0.598.
- Using full-repo context with GPT on three small repos reduces TypeCheck errors but lowers TypeSim on some repos (e.g., private-gpt 0.855 single-file vs 0.601 full repo).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use full repository context instead of single-file context | GPT type inference · gptme | TypeCheck errors: 84 -> 15 | Helped |
| Use full repository context instead of single-file context | GPT type inference · private-gpt | TypeCheck errors: 17 -> 4 | Helped |
| Use full repository context instead of single-file context | GPT type inference · screenshot-to-code | TypeCheck errors: 6 -> 0 | Helped |
| Use full repository context instead of single-file context | GPT type inference · gptme | TypeSim: 0.877 -> 0.966 | Helped |
| Use full repository context instead of single-file context | GPT type inference · private-gpt | TypeSim: 0.855 -> 0.601 | Hurt |
| Use full repository context instead of single-file context | GPT type inference · screenshot-to-code | TypeSim: 0.915 -> 0.696 | Hurt |
| Increase type complexity from depth 1 to depth 5 | LLM type inference · TypyBench types by depth | TypeSim: Average: 0.725 -> 0.598 | Mixed |
| Evaluate on repositories created in 2024 or later rather than before 2024 | LLM type inference · TypyBench repositories by creation date | TypeSim: pre-2024 avg: 0.705; 2024+ avg: 0.649 | Hurt |
| Evaluate rare types rather than all types | LLM type inference · TypyBench repositories | TypeSim | Hurt |
| Use TypeSim instead of exact match for deeper types | Type inference evaluation · TypyBench types by depth | score: -0.026; -0.114; -0.228; -0.289; -0.387 | Helped |
Surprising findings
- Full-repository context reduces type-check errors but can hurt TypeSim, e.g., private-gpt drops from 0.855 to 0.601.
- Models score higher on pre-2024 repositories (0.705) than 2024+ repositories (0.649), suggesting possible data contamination.
Limitations
- Full-repo context evaluated only on three repositories under 64k tokens due to long input and output challenges.
- TypeCheck uses mypy error count as a proxy for consistency and counts only selected error types.
- Evaluation covers only API models and a few small local models; no learning-based methods are trained.
Experimental setup
Datasets: 50 curated Python repositories from GitHub and PyPI
Benchmarks: TypyBench
Metrics: TypeSim TypeCheck (mypy error count) exact match missing rate
Baselines: GPT-4o GPT-4o-mini Claude 3.5 Sonnet DeepSeek-V3 Llama 3 Qwen2.5
Builds on: Typilus PEP 484 mypy
Closest papers by meaning
- Evaluating Program Semantics Reasoning with Type Inference in System $F$ (NeurIPS 2025 · similarity 0.89)
- Dataflow-Guided Neuro-Symbolic Language Models for Type Inference (ICML 2025 · similarity 0.89)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.87)
- CodeSync: Synchronizing Large Language Models with Dynamic Code Evolution at Scale (ICML 2025 · similarity 0.87)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.86)
- Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.