Research map / Architectures and efficiency / Pruning and efficient tuning

Machine Unlearning in Low-Dimensional Feature Subspace

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LOFT, a machine unlearning method that learns a PCA-style projection matrix on penultimate-layer feature covariances of the forgetting and remaining data, plugged into a frozen pretrained model, requiring only one-shot feature fetching.

Problem

Approximate machine unlearning typically requires repeatedly accessing the full training data and updating all pretrained parameters, raising privacy and computational concerns.

Approach

Learn a low-dimensional feature subspace on the Stiefel manifold that captures maximal variance of remaining-data features and minimal variance of forgetting-data features, using only precomputed feature covariance matrices and inserting the projector between backbone and linear head without modifying pretrained weights.

Key results

• On Swin-T Tiny-ImageNet multi-class unlearning, LOFT achieves the smallest average gap to the retrained model (0.85) versus 4.88 for DELETE and 6.34 for BT.
• LOFT optimizes 192,000 parameters (0.69%) with 0.56 s run time versus 27.7M parameters and hundreds of seconds for baselines.
• On ImageNet-1K single-class unlearning with ResNet50, LOFT reports average gap 1.65 with 1.02M parameters (4.01%) and 0.94 s run time, the best among compared methods that use only the forgetting data.

Surprising findings

• LOFT achieves zero MIA gap on instance unlearning yet has a larger test-accuracy drop on remaining data (6.89 points) than some baselines.
• Under single-class unlearning on ImageNet-1K, LOFT's test accuracy on remaining data drops by 6.32 points relative to the retrained model.

Limitations

• Theoretical guarantee is a bound on output difference depending on optimization gap; the authors note it is not rigorous and certified-unlearning frameworks do not directly apply.
• Not applicable to generative models since the pretrained parameters are not updated; generation experiments are only preliminary.
• Less effective when forgetting and remaining data share high similarity, per the remark in the proofs.

Experimental setup

Datasets: Tiny-ImageNet ImageNet-1K

Benchmarks: Tiny-ImageNet ImageNet-1K

Metrics: accuracy on forgetting and remaining data (train/test) membership inference attack success rate average gap to retrained model run time efficiency parameter count

Baselines: FT GA RL SalUn BT L2UL COUN+RL DELETE

Builds on: PCA SalUn Bad Teaching L2UL DELETE

Closest papers by meaning

• Machine Unlearning under Overparameterization (NeurIPS 2025 · similarity 0.89)
• When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.89)
• Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement (NeurIPS 2024 · similarity 0.88)
• Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models (ICLR 2024 · similarity 0.88)
• What makes unlearning hard and what to do about it (NeurIPS 2024 · similarity 0.88)
• LoRA Unlearns More and Retains More (Student Abstract) (AAAI 2025 · similarity 0.87)