Research map / Large language models / Language, safety and interpretability
Multi-Attribute Multi-Grained Adaptation of Pre-Trained Language Models for Text Understanding from Bayesian Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits non-IID information for pre-trained language models from a Bayesian view and proposes M2A, a LoRA/KronA-based multi-attribute multi-grained adapter with multitask text-reconstruction loss and joint distillation, evaluated on FDU-MTL and personalized sentiment datasets.
Problem
Non-IID information (multiple attributes such as category, user, item) is used by neural networks for text understanding, but its effect on pre-trained language models and how to adapt PLMs to it efficiently remains unclear.
Approach
Views each sample through multiple attributes and granularities (coarse IID view plus fine-grained domain modules) as Bayesian posterior samples, averaging low-rank adapters (LoRA for coarse, KronA for fine-grained) and training with an added text-reconstruction (MLM/ARM) likelihood term plus a KL joint-learning loss between coarse and fine-grained models.
Key results
- On FDU-MTL, B-M2A‡ reaches 93.86% average accuracy versus 92.78% for B-MTL and 88.16% for BERT; R-M2A reaches 94.70%.
- On Yelp-2014 accuracy, R-M2A‡ reaches 74.2 versus 69.0 for RoBERTa and 72.1 for R-GNNLM.
- Ablation shows removing text-generation (p(x|w)) loss lowers B-M2A‡ on FDU-MTL from 93.86 to 92.70.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multiple attribute and granularity views (ablation) | M2A · FDU-MTL test | accuracy: 90.98 -> 93.86 | Helped |
| Add multiple attribute and granularity views (ablation) | M2A · IMDB test | accuracy: 52.21 -> 58.70 | Helped |
| Add the fine-grained user view rather than only the fine-grained item view (ablation) | B-M2A‡ · IMDB test | accuracy: 58.43 -> 52.80 | Hurt |
| Add the text generation task (ablation) | B-M2A‡ · FDU-MTL test | accuracy: 92.70 -> 93.86 | Helped |
| Add the text generation task (ablation) | B-M2A‡ · Yelp-2013 test | accuracy: 69.36 -> 72.00 | Helped |
| Add unlabeled data for text generation (ablation) | B-M2A‡ · FDU-MTL test | accuracy: 92.70 -> 93.65 | Helped |
| Use a joint-learning connection of granularities instead of averaging fine-grained modules (ablation) | R-M2A† · IMDB test | accuracy: 52.2 -> 54.2 | Helped |
| Use a joint-learning connection of granularities instead of random fine-grained module selection (ablation) | R-M2A† · Yelp-2013 test | accuracy: 66.7 -> 70.7 | Helped |
| Keep separation training for the coarse-view module (ablation) | R-M2A† · IMDB test | accuracy: 52.8 -> 54.2 | Helped |
| Add the KL loss (ablation) | R-M2A† · Yelp-2013 test | accuracy: 70.5 -> 70.7 | Helped |
| Add the coarse-grained model's multitask loss (ablation) | R-M2A† · Yelp-2014 test | accuracy: 70.1 -> 70.5 | Helped |
| Use LoRA rather than KronA for the coarse-grained view (ablation) | R-M2A · Yelp-2013 dev | accuracy: 68.03 -> 71.18 | Helped |
Surprising findings
- Fine-grained-only (all-user or item) modules without a coarse view perform worse than the coarse baseline on personalized datasets, with the all-view mixture reaching 52.21 on IMDB versus 58.70 for B-M2A‡.
- Full fine-tuning with LoRA on the fine-grained setup ran out of memory, while KronA fine-tuning was feasible.
Limitations
- Experiments are limited to sentiment analysis datasets and a few PLM backbones (BERT, RoBERTa, GPT-2).
- The Bayesian motivation relies on approximations; the paper leaves a detector of data heterogeneity and a multi-view dataset to future work.
- Results are sensitive to the balance factor alpha for the text-generation loss, requiring careful configuration.
Experimental setup
Datasets: FDU-MTL IMDB Yelp-2013 Yelp-2014
Benchmarks: FDU-MTL IMDB Yelp-2013 Yelp-2014
Metrics: Accuracy RMSE Macro-F1
Baselines: BERT DAEA BERTMasker KCL-KB MTL IUPC MAA GS GNNLM RoBERTa GPT2
Builds on: LoRA KronA BitFit Bayesian neural networks
Closest papers by meaning
- BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models (NeurIPS 2024 · similarity 0.86)
- Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning (NeurIPS 2025 · similarity 0.84)
- Text-to-LoRA: Instant Transformer Adaption (ICML 2025 · similarity 0.84)
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning (ICLR 2024 · similarity 0.84)
- Training-Free Bayesianization for Low-Rank Adapters of Large Language Models (NeurIPS 2025 · similarity 0.84)
- LLM Augmented LLMs: Expanding Capabilities through Composition (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.