Research map / Large language models / Retrieval and knowledge

Knowledge Tagging with Large Language Model Based Multi-Agent System

AAAI 2025 · Large language models › Retrieval and knowledge · approach: Explanatory and multiple-choice QA benchmarks · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an LLM-based multi-agent system (planner, question solver, semantic judger, numerical judger with Python code execution) for knowledge tagging of math questions, evaluated on MathKnowCT against embedding, fine-tuned PLM and single-LLM baselines.

Problem

Automating knowledge tagging of questions (binary matching of knowledge definitions to question stems) is hard when definitions have intricate semantic and strict numerical constraints, where single-LLM and PLM methods fall short of human experts.

Approach

Decompose a knowledge definition into independent sub-constraints via a planning agent; route semantic constraints to an LLM judge and numerical constraints to LLM-generated Python code executed to verify them, then AND the results.

Key results

• Multi-agent Mixtral (base) reaches 79.27% accuracy and 68.17% precision, the best base-size result, while the multi-agent GPT-4o large model reaches 86.91% accuracy and 80.47% precision versus 88.38% accuracy for single GPT-4o.
• Multi-agent design improves precision for base-sized LLMs (e.g., Llama-3-8B precision 51.84 to 62.01), but recall drops for multi-agent variants.
• Industrial deployment at Squirrel Ai reportedly saved at least $306,750 in labeling cost (self-reported estimate).

Measured effects reported in the paper

Surprising findings

• Single-LLM GPT-4o achieves higher accuracy (88.38%) and F1 (85.46) than its multi-agent counterpart (86.91%, 81.75), contrary to the claim that the multi-agent design uniformly helps.
• Multi-agent design lowers recall, attributed to errors compounding across intermediate steps.

Limitations

• Recall decreases with the multi-agent pipeline because intermediate-step errors increase false negatives.
• Large-model gains are smaller; GPT-4o single-LLM outperforms the multi-agent variant on accuracy and F1.
• Evaluation limited to one dataset (MathKnowCT, 24 concepts, grades 1-3).

Experimental setup

Datasets: MathKnowCT

Benchmarks: MathKnowCT

Metrics: accuracy precision recall F1-score

Baselines: GPT-Embed (text-embedding-3-small) SBERT BERT fine-tuning RoBERTa fine-tuning T5 fine-tuning single LLM 2-shot CoT inference

Builds on: chain-of-thought prompting in-context learning tool-use LLMs

Closest papers by meaning

• Chain of Agents: Large Language Models Collaborating on Long-Context Tasks (NeurIPS 2024 · similarity 0.86)
• S-DAG: A Subject-Based Directed Acyclic Graph for Multi-Agent Heterogeneous Reasoning (AAAI 2026 · similarity 0.86)
• KARMA: Leveraging Multi-Agent LLMs for Automated Knowledge Graph Enrichment (NeurIPS 2025 · similarity 0.85)
• Knowledge Graph Prompting for Multi-Document Question Answering (AAAI 2024 · similarity 0.85)
• Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.85)
• Synergistic Multi-Agent Framework with Trajectory Learning for Knowledge-Intensive Tasks (AAAI 2025 · similarity 0.84)