Research map / Label-efficient and robust learning / Recognition and long-tail
Preserving Task-Relevant Information Under Linear Concept Removal
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPLINCE, an oblique projection that removes linear predictability of a concept while exactly preserving the covariance of embeddings with a main-task label, and evaluates it on Bias in Bios, multilingual detoxification, Llama profession and Winobias tasks, and CelebA.
Problem
Linear concept-removal methods (e.g., LEACE, SAL) that guarantee linear guardedness of a protected concept also tend to erase task-relevant information correlated with that concept, harming fairness-utility tradeoffs and causal interventions on frozen models.
Approach
Construct the minimum-distortion affine projection whose kernel contains Cov(x,z) (the concept) and whose range contains Cov(x,y) (the task label), giving a closed-form oblique projection SPLINCE that generalizes LEACE by additionally preserving task covariance.
Key results
- On Bias in Bios and Multilingual Text Detoxification, SPLINCE outperforms SAL and LEACE in accuracy and worst-group accuracy as task-concept correlation grows.
- On the Llama 2 7B profession dataset, SPLINCE retains factual coefficient exp(beta_fact)=24.27 versus 12.14 for LEACE and 5.90 for SAL, while reducing stereotype reliance to 0.79.
- On Winobias, SPLINCE improves coreference accuracy over other projections for two of three Llama models, notably on anti-stereotypical prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SPLINCE instead of LEACE | Llama 2 7B on the profession dataset · Profession dataset | exp(beta_fact): 12,14 -> 24,27$^*$ | Helped |
| Apply SPLINCE instead of LEACE | Llama 2 13B on the profession dataset · Profession dataset | exp(beta_fact): 16,32 -> 33,24$^*$ | Helped |
| Apply SPLINCE instead of LEACE | Llama 3 8B on the profession dataset · Profession dataset | exp(beta_fact): 7,68 -> 13,43$^*$ | Helped |
| Apply any of SAL, LEACE, or SPLINCE | Llama models on the profession dataset · Profession dataset | exp(beta_stereo) | Helped |
| Apply SAL or LEACE | Llama models on the profession dataset · Profession dataset | exp(beta_fact) | Hurt |
| Apply SPLINCE instead of other projections | Llama models on Winobias · Winobias | coreference accuracy | Helped |
| Apply SPLINCE instead of other projections | Llama models on anti-stereotypical Winobias prompts · Winobias anti-stereotypical prompts | accuracy | Helped |
| Use SPLINCE instead of SAL or LEACE as task–concept correlation strengthens | Regularized last-layer classification · Bias in Bios and Multilingual Text Detoxification | accuracy | Helped |
| Use SPLINCE instead of SAL or LEACE as task–concept correlation strengthens | Regularized last-layer classification · Bias in Bios and Multilingual Text Detoxification | worst-group accuracy | Helped |
| Use SPLINCE instead of LEACE for a frozen last layer with whitened data | Linear regression · Whitened-data regression | excess risk | Helped |
Surprising findings
- Projections with the same kernel (all linear-guardedness methods) yield identical predictions after unregularized re-training of the last layer, so the range matters only without re-training or with regularization.
- SPLINCE performs relatively worse on vision classification tasks (CelebA, Waterbirds) than on NLP tasks.
Limitations
- SPLINCE prioritizes preserving the task covariance over minimizing embedding distortion, which can produce distortive changes.
- SPLINCE sometimes fails when intervening in middle layers of LMs.
- Theoretical guarantees that task covariance preservation yields main-task performance are lacking; vision results are weaker.
Experimental setup
Datasets: Bias in Bios Multilingual Text Detoxification Winobias CelebA Waterbirds
Benchmarks: Bias in Bios Multilingual Text Detoxification Winobias CelebA Waterbirds
Metrics: accuracy worst-group accuracy odds ratio coefficients coreference accuracy
Baselines: LEACE SAL
Builds on: LEACE SAL INLP RLACE
Closest papers by meaning
- Obliviator Reveals the Cost of Nonlinear Guardedness in Concept Erasure (NeurIPS 2025 · similarity 0.84)
- Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation (CVPR 2025 · similarity 0.84)
- EraseFlow: Learning Concept Erasure Policies via GFlowNet-Driven Alignment (NeurIPS 2025 · similarity 0.83)
- Removing Spurious Concepts from Neural Network Representations via Joint Subspace Estimation (ICML 2024 · similarity 0.83)
- CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Models (NeurIPS 2025 · similarity 0.83)
- Consistency-Preserving Concept Erasure via Unsafe–Safe Pairing and Directional Fisher-weighted Adaptation (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.