Research map / Large language models / Language, safety and interpretability
RealtimeTool: Parallel Decoding for Real-Time LLM Function Calling
Xiaoxin Shi, Jiaxin Wan, Linkang Dong, Wei Jiang, Yue Liu, Zengfeng Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train SimpleTool special-token parallel decoding instead of baseline JSON output | Qwen2.5/Qwen3 models 0.5B-14B · Five benchmarks average | overall accuracy: +2.9% average; 70.9 -> 75.3 (0.5B) | Improved |
| train SimpleTool special-token parallel decoding | Qwen models · Five benchmarks average | function accuracy: +7.1% | Improved |
| train SimpleTool on Mobile Actions | Qwen3-4B · Mobile Actions | overall accuracy: 68.0 -> 84.5 | Improved |
| train SimpleTool on BFCL Live | Qwen2.5-1.5B · BFCL Live | overall accuracy: 67.4 -> 63.6 | Hurt |
| train SimpleTool on BFCL Exec | Qwen3-4B · BFCL Exec | overall accuracy: 94.6 -> 89.9 | Hurt |
| compress low-entropy tokens with special tokens and decode heads in parallel | Qwen models · BFCL-v3 single-turn, vLLM, RTX 4090 | end-to-end speedup: 3.07x to 3.88x | Improved |
| use parallel decoding with 8 heads vs single sequence (ablation) | SimpleTool · Batch scaling across six models | per-token overhead: +8.2% overhead, 93.0% efficiency | No clear effect |
| increase LoRA rank from 64 to 1024 (ablation) | ST-Qwen3-4B · BFCL-v3 | accuracy: 85.5 -> 87.0 | Improved |
| LoRA rank 512 vs 256 (ablation) | ST-Qwen3-4B · BFCL-v3 | accuracy: 86.4 -> 85.3 | Hurt |
| add synthetic augmentation data to xLAM (ablation) | ST-Qwen3-4B · BFCL-v3 | accuracy: 85.3 -> 86.3 | Improved |
| ST-Qwen-0.5B vs FunctionGemma after fine-tuning | edge function calling · Mobile Actions fine-tuned | accuracy: 85.0 -> 86.2 | Improved |
| ST-Qwen-0.5B vs FunctionGemma latency | edge function calling · Mobile Actions latency | P90 latency (ms): 139.5 -> 74.5 | Improved |
Closest papers by meaning
- The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements (NeurIPS 2025 · similarity 0.87)
- Evaluating Test-Time Scaling of General LLM Agents (NeurIPS 2026 · similarity 0.86)
- A Two-Tier Perspective on Inference-Time Parallelism in Multi-Agent LLM Systems (ICML 2026 · similarity 0.86)
- Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning (ICLR 2025 · similarity 0.86)
- SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · similarity 0.86)
- Speculative Actions: A Lossless Framework for Faster AI Agents (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
