Research map / Large language models / Retrieval and knowledge
Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NesyCD, a neural-symbolic distillation method where an LLM analyzes small-model errors to build a symbolic knowledge base that is retrieved during SLM fine-tuning and inference, improving SLMs on complex reasoning.
Problem
Small language models (<=7B) struggle with complex reasoning because they cannot capture sparse, specialized knowledge through standard chain-of-thought distillation from LLMs.
Approach
Split complex-reasoning knowledge into general abilities (distilled into neural SLM weights via CoT) and specialized knowledge (generated by the teacher LLM from the student's error cases and stored in a symbolic KB), then fine-tune the student with retrieved KB knowledge plus auxiliary tasks (answer prediction, direct CoT) and retrieve adaptively using a generation-confidence threshold.
Key results
- On LLaMA2-7B, NesyCD averages 48.7 across seven datasets versus 41.1 for the best CoT distillation baseline (CasCoD) and 39.2 for Std-CoT.
- NesyCD with LLaMA3-8B and Qwen2-7B surpasses GPT-3.5-turbo (63.2 average) per the paper's claims; Qwen2-7B NesyCD averages 69.5 and LLaMA3-8B NesyCD averages 63.2.
- On TinyLLaMA-1.1B, NesyCD averages 29.3 versus 23.3 for KARD (BM25) and 23.2 for Std-CoT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the learning summary to NesyCD (ablation) | NesyCD · BBH, LLaMA2-7B | accuracy (%): 68.48 -> 75.49 | Helped |
| add supplementary knowledge to NesyCD (ablation) | NesyCD · BBH, LLaMA2-7B | accuracy (%): 68.86 -> 75.49 | Helped |
| add augmented distillation (ablation) | NesyCD · BBH, LLaMA2-7B | accuracy (%): 64.34 -> 75.49 | Helped |
| add answer prediction and direct CoT tasks (ablation) | NesyCD · BBH, LLaMA2-7B | accuracy (%): 66.51 -> 75.49 | Helped |
| add specialized knowledge (ablation) | Std-CoT · BBH, LLaMA2-7B | accuracy (%): 58.13 -> 61.24 | Helped |
| add specialized knowledge (ablation) | Zero-shot-CoT · BBH, LLaMA2-7B | accuracy (%): 13.51 -> 14.57 | Helped |
| retrieve 2 knowledge items instead of 1 (ablation) | NesyCD · BBH, LLaMA2-7B | accuracy (%): 75.49 -> 72.41 | Hurt |
| retrieve 3 knowledge items instead of 1 (ablation) | NesyCD · BBH, LLaMA2-7B | accuracy (%): 75.49 -> 73.28 | Hurt |
| replace Contriever with DPR (ablation) | NesyCD · BBH, LLaMA2-7B | accuracy (%): 75.49 -> 74.47 | Hurt |
| replace Contriever with BM25 (ablation) | NesyCD · BBH, LLaMA2-7B | accuracy (%): 75.49 -> 75.55 | Mixed |
Surprising findings
- Increasing the number of retrieved knowledge items at inference (m=2, 3) lowers performance relative to m=1.
- CasCoD performs far below Std-CoT on LLaMA3-8B and Qwen2 models in the extended results (e.g., 34.5 average on LLaMA3-8B), suggesting its gains do not transfer across backbones.
Limitations
- Effectiveness depends on the quality of teacher-generated knowledge, which remains far from gold-standard knowledge.
- Teacher limited to GPT-3.5-turbo due to budget; GPT-4 not tested.
- Only models below 7B-class mainstream families tested; larger 13B/72B students not explored.
Experimental setup
Datasets: MMLU MetaMathQA BBH GSM8K ARC BB AGIEval GSM8K-PLUS Wikipedia
Benchmarks: BBH GSM8K BB-sub AGIEval GSM8K-PLUS ARC-E ARC-C
Metrics: accuracy (exact match)
Baselines: Zero-shot Zero-shot-CoT Fine-tuning Knowledge-Augmented Fine-tuning Std-CoT MT-CoT Step-by-step KARD (BM25) CasCoD
Builds on: Std-CoT KARD CasCoD Step-by-step distillation LoRA
Closest papers by meaning
- Position: The Road to Generalizable Neuro-Symbolic Learning Should be Paved with Foundation Models (NeurIPS 2026 · similarity 0.88)
- Logically Consistent Language Models via Neuro-Symbolic Integration (ICLR 2025 · similarity 0.87)
- Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.87)
- Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation (NeurIPS 2026 · similarity 0.87)
- RLKD: Distilling LLMs’ Reasoning via Reinforcement Learning (AAAI 2026 · similarity 0.87)
- Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.