Research map / Label-efficient and robust learning / Recognition and long-tail
Exploring and Leveraging Class Vectors for Classifier Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Class Vectors, per-class latent representation shifts between pretrained and fine-tuned encoders, used for training-free latent steering or weight-space mapping to edit classifiers (unlearning, snow adaptation, typographic-attack defense, backdoor trigger optimization).
Problem
Post hoc classifier editing (forgetting specific classes, adapting to distribution shifts) is costly or limited to correcting image-wise errors; existing methods need retraining or auxiliary masks and are underexplored in image classification.
Approach
Define a Class Vector as the difference between fine-tuned and pretrained penultimate-layer class centroids; justify its linearity via cross-task linearity and its independence via neural collapse, then use it to steer latents or map it into encoder weights for class-level edits and arithmetic.
Key results
- Class unlearning on ViT-B/16 drives forget-class accuracy to 0.0 on MNIST, EuroSAT and GTSRB while keeping retained accuracy at 99.7, 99.5 and 98.6 (Class Vector), outperforming Retrained and NegGrad.
- Snowy ImageNet adaptation: Class Vector^† reaches 72.7 to 78.3 average accuracy across ViT-B/16, B/32, L/14 versus 55.2 to 60.2 for pretrained, and is competitive with DirMatch.
- Typographic attack defense with ViT-B/32 reaches 98.9 average accuracy for Class Vector versus 76.7 for the attacked pretrained model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Class Vector steering instead of retraining for class unlearning | ViT-B/16 classifier · MNIST | ACC_f: 99.9{ $\pm$0.1} -> 0.0{ $\pm$0.0} | Helped |
| use Class Vector steering instead of retraining for class unlearning | ViT-B/16 classifier · MNIST | ACC_r: 76.4{ $\pm$0.0} -> 99.7{ $\pm$0.0} | Helped |
| map the editing vector into weight space rather than using latent-space steering | Class Vector on ViT-B/16 · EuroSAT | ACC_r: 99.5{ $\pm$0.2} -> 99.7{ $\pm$0.0} | Helped |
| map the editing vector into weight space rather than using latent-space steering | Class Vector on ViT-B/16 · GTSRB | ACC_r: 98.6{ $\pm$0.0} -> 93.4{ $\pm$0.0} | Hurt |
| use Class Vector editing instead of the pretrained classifier | ViT-B/16 classifier · Snowy ImageNet | Average accuracy: 55.2{ $\pm$24.6} -> 69.7{ $\pm$2.6} | Helped |
| use mapped Class Vector editing instead of DirMatch | ViT-B/32 classifier · Snowy ImageNet | Average accuracy: 73.9{ $\pm$23.5} -> 76.2{ $\pm$21.2} | Helped |
| use mapped Class Vector editing instead of DirMatch | ViT-B/16 classifier · Typo Attack | Average accuracy: 97.7{ $\pm$3.1} -> 98.9{ $\pm$2.5} | Helped |
| use Class Vector steering instead of DirMatch | ViT-B/32 classifier · Typo Attack | Average accuracy: 91.1{ $\pm$8.3} -> 98.9{ $\pm$2.5} | Helped |
| use Class Vector steering instead of random-vector editing | ViT-B/16 classifier · EuroSAT unlearning | ACC_r: 80.9{ $\pm$26.1} -> 99.5{ $\pm$0.2} | Helped |
| use Class Vector steering instead of retraining with limited samples | classifier editor · Snowy ImageNet | Average accuracy | Mixed |
Surprising findings
- Class Vectors achieve near-zero forget accuracy on several unlearning tasks, yet the latent-steering variant retains high accuracy without any training.
Limitations
- Assumes class representations are well-structured and approximately linearly separable; may fail with high intra-class variance, noisy labels, or long-tailed data.
- Only single-label classification is studied; multi-label or hierarchical settings remain open.
- Latent steering needs access to class centroids and weight mapping needs partial access to model internals, limiting closed-source use.
Experimental setup
Datasets: MNIST EuroSAT SVHN GTSRB RESISC45 DTD CIFAR-10 CIFAR-100 ImageNet Snowy ImageNet web-sourced typographic attack images AG-NEWS
Benchmarks: MNIST EuroSAT GTSRB RESISC45 DTD Snowy ImageNet typographic attack images MNIST (ResNet/ConvNeXt) AG-NEWS DBPedia-14 20-Newsgroups SVHN
Metrics: accuracy on forget class (ACC_f) accuracy on retained classes (ACC_r) average accuracy attack success rate clean accuracy
Baselines: Retrained NegGrad Random Vector DirMatch BadNet Task Vector
Builds on: task vectors cross-task linearity neural collapse DirMatch linear mode connectivity
Closest papers by meaning
- Generative Classifiers Avoid Shortcut Solutions (ICLR 2025 · similarity 0.84)
- DisCEdit: Model Editing by Identifying Discriminative Components (NeurIPS 2024 · similarity 0.83)
- Adversarial Representation Engineering: A General Model Editing Framework for Large Language Models (NeurIPS 2024 · similarity 0.82)
- CAT: Exploiting Inter-Class Dynamics for Domain Adaptive Object Detection (CVPR 2024 · similarity 0.82)
- Cross-Class Feature Augmentation for Class Incremental Learning (AAAI 2024 · similarity 0.82)
- Class-Attribute Priors: Adapting Optimization to Heterogeneity and Fairness Objective (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.