Research map / Large language models / Instruction and fine-tuning
Fundamental Limitations in Pointwise Defences of LLM Finetuning APIs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that pointwise (per-sample) detectors on fine-tuning APIs can be evaded by covert attacks that encode harmful MCQ answers as benign, low-perplexity variations in model output (classify and flower attacks), succeeding against OpenAI's fine-tuning API.
Problem
Fine-tuning APIs defend against misuse with pointwise detectors that screen individual training samples and inference interactions; prior covert attacks (cipher/steganography) are easily flagged as suspicious, so the question is whether pointwise defences can be fundamentally bypassed.
Approach
Repurpose systematic, benign, low-perplexity variations in a model's responses to benign-transformed harmful queries (e.g. classify-as-malicious phrasings, or a random flower name) as an answer-encoding channel; each training and inference sample is individually benign and likely under the pre-finetuned model, so pointwise monitors cannot flag them.
Key results
- On IED-MCQ with GPT-4o and no monitor, the classify and flower attacks reach 98.4% correct versus 90.2% for CMT-EndSpeak and 62.3% for CMT-Walnut.
- Under an o1-mini suspiciousness monitor, classify keeps 96.7% (IED-MCQ) and 100% (Copyright-MCQ) correct, while CMT-Walnut drops to 16.4% and 28.2% correct.
- Distribution-level detection with o3 flags classify attack datasets 76.0% of the time at N=15, while o4-mini fails to separate the datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Map answer choices to benign classification responses instead of learning an input/output cipher | GPT-4o fine-tuning attack · IED-MCQ test, no monitor | accuracy: 62.3% -> 98.4% | Helped |
| Map answer choices to benign classification responses instead of poetic-phrase steganography | GPT-4o fine-tuning attack · IED-MCQ test, no monitor | accuracy: 90.2% -> 98.4% | Helped |
| Use a pointwise model-spec violation monitor | classify attack · IED-MCQ and Copyright-MCQ | accuracy: 98.4% and 100.0% | No clear effect |
| Use a suspicious-interaction monitor | classify attack · IED-MCQ and Copyright-MCQ | accuracy: 98.4% -> 96.7%; 100.0% -> 100.0% | Mixed |
| Use a suspicious-interaction monitor | flower attack · IED-MCQ and Copyright-MCQ | accuracy: 98.4% -> 82.0%; 100.0% -> 87.2% | Hurt |
| Train the variation mapping on benign Commonsense-QA questions only, with no harmful text in training (ablation) | classify attack · IED-MCQ and Copyright-MCQ | accuracy: IED-MCQ: 98.4% -> 96.7%; Copyright-MCQ: 100.0% -> 69.2% | Mixed |
| Train the variation mapping on benign Commonsense-QA questions only, with no harmful text in training (ablation) | flower attack · IED-MCQ and Copyright-MCQ | accuracy: IED-MCQ: 98.4% -> 98.4%; Copyright-MCQ: 100.0% -> 79.5% | Mixed |
| Apply the suspicious-interaction monitor (ablation) | flower-benign attack · IED-MCQ test | accuracy: 98.4% -> 78.7% | Hurt |
| Use classify encoding and prompting after fine-tuning | GPT-4o · BSD-Cyber | accuracy: 0.573 -> 0.467 | Hurt |
| Use flower encoding and prompting after fine-tuning | GPT-4o · BSD-Cyber | accuracy: 0.573 -> 0.587 | No clear effect |
| Use classify encoding and prompting after fine-tuning | GPT-4o · GPQA | accuracy: 0.428 -> 0.406 | Hurt |
| Use flower encoding and prompting after fine-tuning | GPT-4o · GPQA | accuracy: 0.428 -> 0.364 | Hurt |
Surprising findings
- Training only on benign Commonsense-QA questions (no harmful text in training) still yields 96.7% correct on IED-MCQ for classify-benign.
- Sonnet 3.5 can decode the CMT ciphers in context, so a stronger monitor flags them, whereas the pointwise o1-mini monitor largely did not.
Limitations
- Experiments are restricted to multiple-choice questions on proxy harmful datasets that are synthetic and may contain low-quality samples.
- Attacks evade only stateless pointwise detectors; distribution-level monitors (o3) can catch them at N=15 and the authors note mixing in benign data could defeat that defence.
- Free-form extension is only theoretical or a small SimpleQA uplift experiment (7.2% to 12.6%).
Experimental setup
Datasets: IED-MCQ Copyright-MCQ Alpaca-GPT4 Commonsense-QA XSTest OR-Bench GPQA BSD-Cyber SimpleQA
Benchmarks: IED-MCQ Copyright-MCQ GPQA BSD-Cyber SimpleQA Commonsense-QA
Metrics: attack success rate (correct %) blocked rate refusal rate accuracy
Baselines: CMT-Walnut CMT-EndSpeak
Builds on: covert malicious fine-tuning semantic entropy OpenAI fine-tuning API moderation
Closest papers by meaning
- Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs (NeurIPS 2025 · similarity 0.89)
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation (NeurIPS 2025 · similarity 0.89)
- Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation (ICML 2024 · similarity 0.88)
- Representation Noising: A Defence Mechanism Against Harmful Finetuning (NeurIPS 2024 · similarity 0.88)
- Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety (ICML 2025 · similarity 0.87)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.