Research map / Large language models / Agents, code and math
Can Large Language Models Derive High-Level Cognition from Low-Level and Fragmented Foundational Information?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces High-Level Cognition (HLC), a task of deriving high-level soccer playing-style conclusions from low-level match commentary, builds the MatchIntel dataset with three question formats, and shows that QLoRA fine-tuning of Llama3-8B-Instruct (TraitsMind) beats near-random LLM baselines.
Problem
Whether LLMs can derive high-level conclusions (team playing styles) from low-level, fragmented foundational information such as soccer match commentary, which requires integrated understanding, synthesizing, and reasoning without an existing chain-of-thought framework.
Approach
Frames HLC as an integrated understanding-synthesizing-reasoning task, evaluates it with MCQ, SCQ, and TFQ formats built from soccer commentary and playing-style labels, sets a random-chance threshold (expected accuracy plus 3 standard deviations), and shows QLoRA fine-tuning of Llama3-8B-Instruct raises accuracy above that threshold.
Key results
- Closed and open LLMs perform near the random threshold (e.g., GPT-3.5-Turbo 0.106 MCQ vs 0.118 threshold; Llama3-70B 0.288 SCQ vs 0.291 threshold).
- Fine-tuned TraitsMind (Llama3-8B-Instruct + QLoRA) reaches 0.135 MCQ, 0.431 SCQ, 0.653 TFQ, improving over random-level by 14.4%, 48.1%, and 19.4%.
- Accuracy of TraitsMind drops as temperature increases, while other models show no clear trend.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama3-Instruct 8B on MatchIntel | Llama3-Instruct 8B · MatchIntel MCQ | accuracy: 0.111 -> 0.135 | Helped |
| Fine-tune Llama3-Instruct 8B on MatchIntel | Llama3-Instruct 8B · MatchIntel SCQ | accuracy: 0.267 -> 0.431 | Helped |
| Fine-tune Llama3-Instruct 8B on MatchIntel | Llama3-Instruct 8B · MatchIntel TFQ | accuracy: 0.519 -> 0.653 | Helped |
| Use Llama3-Instruct 8B instead of Llama2-Chat 7B as the foundation model (ablation) | Fine-tuned Llama foundation models · MatchIntel MCQ | accuracy: 0.125 -> 0.135 | Helped |
| Use Llama3-Instruct 8B instead of Llama2-Chat 7B as the foundation model (ablation) | Fine-tuned Llama foundation models · MatchIntel SCQ | accuracy: 0.324 -> 0.431 | Helped |
| Use Llama3-Instruct 8B instead of Llama2-Chat 7B as the foundation model (ablation) | Fine-tuned Llama foundation models · MatchIntel TFQ | accuracy: 0.596 -> 0.653 | Helped |
| Increase inference temperature (ablation) | TraitsMind · MatchIntel | accuracy | Hurt |
| Increase inference temperature (ablation) | Other tested LLMs · MatchIntel | accuracy | No clear effect |
| Reduce quantization bit-width (ablation) | Fine-tuned Llama models · MatchIntel | accuracy | Hurt |
Surprising findings
- Larger models do not outperform smaller ones on HLC: Llama3-8B-Instruct beats Llama3-70B on MCQ and TFQ.
- Current state-of-the-art LLMs, including GPT-3.5-Turbo and Gemini-Pro, perform at random-chance level on the soccer playing-style tasks.
Limitations
- Fine-tuning used only one set of initial hyperparameters; the highest achievable level was not explored.
- LLM accuracy is highly sensitive to prompt wording, and prompts were only a preliminary design.
- Evaluation is limited to a single domain (soccer) and three question formats.
Experimental setup
Datasets: MatchIntel
Benchmarks: MatchIntel
Metrics: accuracy
Baselines: GPT-3.5-Turbo Gemini-Pro Zephyr-Beta Gemma-1.1-Instruct Llama3-Instruct 8B Llama3-Instruct 70B Mixtral-Instruct v0.1 DBRX-Instruct
Builds on: QLoRA LoRA Llama 3
Closest papers by meaning
- Generating Computational Cognitive models using Large Language Models (NeurIPS 2025 · similarity 0.90)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.89)
- Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study (NeurIPS 2025 · similarity 0.88)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.88)
- Who Reasons in the Large Language Models? (NeurIPS 2025 · similarity 0.88)
- Exploring the LLM Journey from Cognition to Expression with Linear Representations (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.