Research map / Large language models / Language, safety and interpretability

AC-LoRA: (Almost) Training-Free Access Control Aware Multi-Modal LLMs

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AC-LoRA, a training-free access-control-aware system that retrieves permitted per-domain LoRA adapters by query similarity and mixes them by similarity weights, for text and multimodal LLMs.

Problem

Corporate LLM chatbots leak sensitive information via memorization, and strict per-user access control is hard to enforce; training separate models per permission set scales as 2^n, and existing MoE-style LoRA mixing needs trained routers that may expose restricted data.

Approach

Fine-tune isolated LoRAs per permission zone, store their document embeddings in a vector DB, retrieve top-k LoRAs by query similarity intersected with the user's permission vector, and merge the retrieved LoRA outputs weighted by normalized similarity scores, with no trained gate; non-permitted high-similarity LoRAs are surfaced as hints.

Key results

• On FlanV2, AC-LoRA (k=3, fetch_k=10) reaches 61.7 Rouge-1 on struct-to-text and 65.0 on commonsense, matching or exceeding LoRARetriever on 8 of 10 domains as claimed.
• Retriever achieves above 90% accuracy in retrieving the correct LoRA on RepLiQA while retrieving at most 3 LoRAs.
• Loading the 8B model with 17 LoRAs (r=64) needs about 40GB GPU memory without quantization; AC-LoRA TTFT is lower than RAG-like long-context Llama3.1-8B at 5K and 10K contexts.

Measured effects reported in the paper

Surprising findings

• Combining LoRAs improves over the weaker LoRA but generally performs worse than the higher-performing single LoRA, with only 7.71% of queries improving over both.
• Memorization amplifies at higher sampling temperature, leaking many verbatim 12-gram substrings from fine-tuned LoRA training data.

Limitations

• Relies on LLM capacity to integrate knowledge during fine-tuning; combining knowledge across LoRAs is only preliminarily studied.
• Hinting mechanism can enable membership-inference-like attacks exposing existence of confidential datasets.
• Frequent LoRA swapping in memory-constrained settings significantly increases time to first token; multi-modal evaluation is only a proof of concept.

Experimental setup

Datasets: RepLiQA FlanV2 WikiArt CS-Combi (derived from RepLiQA) arXiv confidential and quantum computing papers

Benchmarks: RepLiQA FlanV2 WikiArt

Metrics: grade on 0-5 scale ROUGE-1/2/L BLEU exact match retrieval accuracy time-to-first-token latency memory

Baselines: LoRARetriever MoE Top1 MoE Top3 MoE Soft SMEAR Adapter Soup LoRA Hub Perfect Selection

Builds on: LoRA LoRARetriever MoLE RAG

Closest papers by meaning

• Permissioned LLMs: Enforcing Access Control in Large Language Models (NeurIPS 2025 · similarity 0.86)
• DOMBA: Double Model Balancing for Access-Controlled Language Models via Minimum-Bounded Aggregation (AAAI 2025 · similarity 0.85)
• Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives (NeurIPS 2024 · similarity 0.85)
• Unified Parameter-Efficient Unlearning for LLMs (ICLR 2025 · similarity 0.84)
• PrivAuditor: Benchmarking Data Protection Vulnerabilities in LLM Adaptation Techniques (NeurIPS 2024 · similarity 0.84)
• Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.83)