Research map / Label-efficient and robust learning / Recognition and long-tail

Advancing Personalized Learning with Neural Collapse for Long-Tail Challenge

ICML 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Classifier rebalancing and margin losses

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes NCAL, which adds a text-modality collapse regularizer to LoRA fine-tuning of LLMs so text embeddings approach a simplex ETF, improving classification on long-tail personalized-learning datasets (TMWPL, PMTD).

Problem

Data-based personalized learning benchmarks in education are long-tailed and class-imbalanced, which degrades LLM classification performance; prior work assumes high-quality balanced annotations.

Approach

Extends neural collapse (simplex ETF geometry of features) from vision classifiers to text-based LLM representations and adds a TC loss that penalizes pairwise inner products of different-class text embeddings during LoRA fine-tuning, claimed to counter imbalance-driven gradient bias.

Key results

• NCAL-Qwen2.5-Instruct 7B reaches 74.86 Acc / 75.58 Macro-F1 on TMWPL vs 61.14 / 60.59 for the baseline Qwen2.5-7B and 67.43 for Vicuna-13B.
• NCAL-Llama3.1-Instruct 8B reaches 67.50 Acc / 64.92 Macro-F1 on PMTD vs 61.88 Acc for Vicuna-1.5-7B baseline (Llama3.1 baseline 65.62 Acc).
• Ablation: removing NCAL from Qwen2.5-7B-Instruct drops PMTD accuracy from 70.31 to 62.50 (prompt removed too).

Measured effects reported in the paper

Surprising findings

• Reasoning-specialized math models excel at math problem classification while instruction-tuned models do better on dialogue classification.
• Base (non-instruct) models still generate uncontrolled outputs even after LoRA fine-tuning on the dataset.

Limitations

• Evaluated only on two in-house education datasets, mostly Chinese math content.
• Theoretical analysis is heuristic gradient reasoning rather than a formal guarantee; hypotheses are stated rather than proven.
• Single run results reported without variance; small test sets (40-50 per class).

Experimental setup

Datasets: TMWPL PMTD

Benchmarks: TMWPL PMTD

Metrics: accuracy macro-F1

Baselines: Qwen2.5-Instruct 7B Qwen2.5-Instruct 14B DeepSeek-V2-Lite-Chat Vicuna1.5 7B Vicuna1.5 13B Llama3.1-Instruct 8B Qwen2.5-Math-Instruct 7B Deepseek-Math-Instruct 7B

Builds on: neural collapse LoRA LLaMA-Factory

Closest papers by meaning

• Linguistic Collapse: Neural Collapse in (Large) Language Models (NeurIPS 2024 · similarity 0.86)
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.85)
• Rethinking Personalization in Large Language Models at the Token Level (ICML 2026 · similarity 0.84)
• Strong Model Collapse (ICLR 2025 · similarity 0.84)
• Escaping Collapse: The Strength of Weak Data for Large Language Model Training (NeurIPS 2025 · similarity 0.84)
• Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders (ICLR 2024 · similarity 0.84)