Research map / Large language models / Agents, code and math
Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic Empirical Study
Yuqi Zhu, Yi Zhong, Jintian Zhang, Ziheng Zhang, Shuofei Qiao, Yujie Luo, Lun Du, Da Zheng, Ningyu Zhang, Huajun Chen
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide table information (columns, types, samples) in the prompt (ablation) | Qwen2.5-7B prompting · QRData | accuracy (%): 6.57 -> 7.54 | Improved |
| provide table information in the prompt (ablation) | Qwen2.5-14B prompting · DiscoveryBench | accuracy (%): 0.42 -> 0.00 | Hurt |
| add irrelevant extra data files to the input (ablation) | Qwen2.5-7B prompting · QRData | accuracy (%): 37.96 -> 34.55 | Hurt |
| add irrelevant extra data files to the input (ablation) | Qwen2.5-14B prompting · QRData | accuracy (%): 52.55 -> 52.07 | No clear effect |
| use code-specialized model (Qwen2.5-7B-Coder) instead of general instruct (ablation) | Qwen2.5-7B · QRData | accuracy (%): 39.71 -> 36.50 | No clear effect |
| use R1-distilled 7B model instead of instruct (ablation) | Qwen2.5-7B · QRData | accuracy (%): 39.71 -> 30.41 | Hurt |
| train on medium-length interaction turns rather than mixed turn lengths (ablation) | Qwen2.5-7B SFT · QRData and DiscoveryBench | accuracy | Improved |
| train on all 5613 samples instead of the medium-turn subset (ablation) | Qwen2.5-7B SFT · QRData | accuracy (%): 49.15 -> 48.66 | Hurt |
| replace original reasoning with full R1 think traces (ablation) | Qwen2.5-7B SFT · DiscoveryBench | accuracy | Hurt |
| replace original reasoning with summarized reasoning (ablation) | Qwen2.5-7B SFT · QRData and DiscoveryBench | accuracy | Improved |
| increase per-turn token budget (ablation) | Qwen2.5-7B SFT · DiscoveryBench | accuracy | Mixed |
| train on medium plus hard difficulty data (ablation) | Qwen2.5-7B SFT · DiscoveryBench | accuracy (%): 18.83 -> 23.01 | Improved |
Closest papers by meaning
- At Which Training Stage Does Code Data Help LLMs Reasoning? (ICLR 2024 · similarity 0.89)
- ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning (NeurIPS 2025 · similarity 0.88)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.88)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.88)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.87)
- To Code or Not To Code? Exploring Impact of Code in Pre-training (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
