Research map / Large language models / Instruction and fine-tuning

StringLLM: Understanding the String Processing Capability of Large Language Models

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Calibration and knowledge in LLM tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces StringLLM, a code-driven pipeline that builds StringBench (1,511 string-processing tasks across multilingual, hash, and random-string datasets), benchmarks LLMs with raw, CoT, and PoT prompting, and shows LLMs lag humans and that PoT-style SFT on Llama-3.1-8B, Gemma-2-9b, and Mistral-7B greatly improves accuracy.

Problem

LLMs struggle with basic string processing (indexing, searching, counting, manipulation), yet this capability has only been studied through isolated case studies without comprehensive benchmarks, root-cause analysis, or effective remedies.

Approach

Use GPT-4o only for code-level operations (composing atomic Python string functions into composite tasks, summarizing code into question templates) and compute ground-truth answers by executing Python, yielding a verifiable benchmark; then fine-tune open LLMs with LoRA on PoT-style solutions to boost string processing.

Key results

• GPT-4o with raw instructions reaches only 45.31% average accuracy on StringBench (PoT 62.08%), while human Python-assisted performance is 98.24%.
• PoT-style LoRA fine-tuning raises average string-processing accuracy by at least 38.80 points (e.g., Mistral-7B-v0.3 Random String +60.11), with Mistral-7B reaching 83.24/90.28/74.18 on Multilingual/Hash/Random.
• Fine-tuned models lose at most 1.35% average on MMLU/Hellaswag/ARC and at most 1.05% on code benchmarks.

Measured effects reported in the paper

Surprising findings

• Inserting spaces between characters improves LLM accuracy, while encoding each character as a separate token ID degrades performance relative to the original tokenizer.
• Random strings are the hardest category, and 8 of 10 LLMs perform worst on them with raw prompts.

Limitations

• Fine-tuning approach is described by the authors as naive and not the primary focus.
• Tokenizer-level alternatives (character-level tokenization) were not successfully pursued due to compute constraints.
• Benchmark relies on GPT-4o for task and template generation, which may bias task distribution.

Experimental setup

Datasets: Flores-200 Alpaca-GPT-4 Dolly-15k Code Alpaca OpenCoder StringBench

Benchmarks: StringBench MMLU Hellaswag ARC HumanEval HumanEval+ MBPP MBPP+

Metrics: accuracy acc-norm

Baselines: GPT-4o GPT-4-Turbo GPT-3.5 DeepSeek-Coder DeepSeek-Chat Gemma-2-9b Mistral-7B-v0.3 Llama-3.1-8B

Builds on: chain-of-thought program of thought LoRA LlamaFactory

Closest papers by meaning

• Needle Threading: Can LLMs Follow Threads Through Near-Million-Scale Haystacks? (ICLR 2025 · similarity 0.87)
• SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator (ICML 2025 · similarity 0.87)
• Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.87)
• Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.87)
• Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.86)
• LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.86)