Research map / Large language models / Agents, code and math
Can LLMs Implicitly Learn Numeric Parameter Constraints in Data Science APIs?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically tests whether LLMs implicitly learn numeric parameter constraints of PyTorch and NumPy APIs across full-program, all-parameter, and single-parameter settings, and releases the DSEVAL benchmark of 19,600 SMT-verified problems.
Problem
Prior DS code generation work assumes LLMs implicitly model numeric parameter constraints in DS library APIs from training data, but this has not been systematically verified.
Approach
Generates controlled problems with SMT-encoded API constraints (via NNSmith) and validates LLM-predicted parameters with Z3, varying input rank and dimension magnitude to measure how accuracy degrades with difficulty, and packages the results as the DSEVAL benchmark.
Key results
- GPT-4-Turbo achieves 57.5% accuracy on hard PyTorch problems in DSEVAL, versus 39.9% for the best open-source model (StarCoder-15b at 39.9%, CodeQwen1.5 at 25.2%)
- On Conv2d groups with the largest value range, GPT-4-Turbo solves only about 24% of problems while other models fall below 14%
- Fold's k_size constraint yields zero valid full programs for DeepSeek Coder-33b and accuracy near 0% at high difficulty
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4-Turbo instead of the strongest open-source model | LLM API-constraint prediction · DSEVAL PyTorch hard | accuracy: 39.2 -> 57.5 | Helped |
| Increase difficulty from easy to hard | GPT-4-Turbo · DSEVAL PyTorch | accuracy: 77.2 -> 57.5 | Hurt |
| Increase difficulty from easy to hard | GPT-4-Turbo · DSEVAL NumPy | accuracy: 95.3 -> 71.4 | Hurt |
| Use chain-of-thought prompting and prohibit groups=1 | GPT-4-Turbo on Conv2d · Conv2d groups prediction, input values up to 255 | accuracy: over 85% | Helped |
| Use chain-of-thought prompting | GPT-4-Turbo on Fold · PyTorch Fold, later difficulty settings | accuracy: less than 5% | Hurt |
| Remove reshape outputs that use the -1 inferred dimension (ablation) | GPT-4-Turbo on PyTorch reshape · PyTorch reshape, individual parameter prediction | accuracy | Hurt |
| Increase the input-dimension value range | LLMs predicting Conv2d groups · PyTorch Conv2d, hardest problems | accuracy: ∼24% | Hurt |
| Increase input rank | LLMs predicting Fold k_size · PyTorch Fold | accuracy: nearly 0% | Hurt |
Surprising findings
- GPT-4-Turbo's high reshape accuracy largely came from predicting -1, which lets the library infer the shape; excluding -1 drops it sharply
- Even GPT-4-Turbo fails on the MaxPool2d padding constraint at high value ranges, while DeepSeek Coder-6.7b handles it by predicting kernel_size//2
Limitations
- Restricted to single-API-call programs and numeric constraints only, ignoring string and other parameter types
- Templates cover 12 APIs in the benchmark out of 28 studied, with fixed value ranges and default parameters
Experimental setup
Datasets: NNSmith constraint encodings DSEVAL
Benchmarks: DSEVAL custom full-program, all-parameter, and single-parameter settings on 28 PyTorch and NumPy APIs
Metrics: accuracy (validity) unique valid rate pass@1 diversity (Hellinger-based for fixed-set APIs)
Baselines: GPT-4-Turbo DeepSeek Coder CodeLlama StarCoder CodeQwen1.5
Builds on: NNSmith DS-1000 chain-of-thought ReAct
Closest papers by meaning
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.88)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.87)
- Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.87)
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.86)
- Is Programming by Example Solved by LLMs? (NeurIPS 2024 · similarity 0.86)
- The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.