Research map / Large language models / Language, safety and interpretability
Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper derives closed-form geometric descriptions of LLM multi-head attention (Minkowski sums of convex hulls) and MLP spline partitions, then uses these to bypass RLHF via prompt-induced intrinsic-dimension increases and to build 7 unsupervised spline features per layer for toxicity and domain detection.
Problem
Understanding LLM internal representations geometrically, and exploiting that geometry for practical tasks such as RLHF robustness and toxicity detection without labeled supervision or heavy fine-tuning.
Approach
Characterize MHA outputs as Minkowski sums of causal convex hulls whose intrinsic dimension is the count of attended preceding tokens, and treat the MLP as a continuous piecewise affine spline; then extract simple region-membership and distance-to-boundary statistics from any pretrained LLM as features for classification.
Key results
- Spline-Llama2-7B with linear classifier reaches 99.18% ROC-AUC on Omni-Toxic, versus 82.08% for the best listed baseline (s-nlp).
- Spline-Llama2-7B with features from first 3 layers plus random forest achieves 94.68% ROC-AUC at about 0.005 s per sample.
- Spline features with gradient-boosted trees reach 99.88% mean ROC-AUC on the Jigsaw test set, versus 98.85% for the Kaggle SOTA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use all spline features with a linear classifier instead of the best listed toxicity baseline | Omni-Toxic toxicity detection · Omni-Toxic | ROC-AUC: 82.08 -> 99.18 | Helped |
| Use spline features from the first 3 layers with a random forest (ablation) | Llama2-7B toxicity detection · Omni-Toxic | ROC-AUC: 94.68 | Helped |
| Use spline features with gradient-boosted trees instead of the Kaggle SOTA system | Jigsaw toxicity detection · Jigsaw Kaggle official test set | ROC-AUC: 98.85% -> 99.88% | Helped |
| Increase the labeled training share from 2% to 10% (ablation) | Spline-Llama2-7B linear classifier · Jigsaw challenge, semi-supervised | ROC-AUC avg.: 77.62 -> 86.30 | Helped |
| Increase the labeled training share from 2% to 10% (ablation) | Spline-Mistral-7B linear classifier · Jigsaw challenge, semi-supervised | ROC-AUC avg.: 74.04 -> 86.09 | Helped |
| Reduce the labeled training share to 1% (ablation) | Spline-Llama2-7B logistic regression · Domain classification, GitHub | ROC-AUC: 99.76% -> 98.60% | Hurt |
| Reduce the labeled training share to 1% (ablation) | Spline-Llama2-7B logistic regression · Domain classification, DollyQA | ROC-AUC: 99.63% -> 98.18% | Hurt |
| Prepend related sentences that increase prompt intrinsic dimension | Llama-2 toxic generation · Toxigen prompts | toxic generation | Helped |
| Prepend unrelated sentences without increasing intrinsic dimension | Llama-2 toxic generation · Toxigen prompts | toxic generation | No clear effect |
| Use random-token prompts that increase intrinsic dimension | Mixtral 8×7B Instruct toxic generation · Toxigen base prompts | RLHF bypass success | Helped |
Surprising findings
- Random tokens prepended to toxic prompts can bypass RLHF refusals when they raise the intrinsic dimension of the input.
- Features flagged many Jigsaw 'clean' labeled prompts as toxic, suggesting label noise in the dataset.
Limitations
- Features are highly aggregated, so the approach is likely suited only to coarse tasks such as toxicity or domain detection, not fine-grained language modeling.
- The theory's partition and bounds are upper bounds (necessary but not sufficient for expressivity), and the ID threshold of 0.1 times max attention is empirically chosen.
Experimental setup
Datasets: The Pile Dolly Q&A Jigsaw toxic comment challenge Toxigen Omni-Toxic (constructed)
Benchmarks: Omni-Toxic Jigsaw Kaggle challenge Pile domain classification
Metrics: ROC-AUC latency (s) percentage of RLHF bypass success
Baselines: martin-ha ToxRoberta nicholasKluge unitary s-nlp citizenlab
Builds on: RoPE Llama 2 Mistral Jigsaw toxic comment challenge Toxigen
Closest papers by meaning
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.88)
- Redefining Experts: Interpretable Decomposition of Language Models for Toxicity Mitigation (NeurIPS 2025 · similarity 0.88)
- When Bad Data Leads to Good Models (ICML 2025 · similarity 0.87)
- Learning Safety Constraints for Large Language Models (ICML 2025 · similarity 0.87)
- Large Language Models can Become Strong Self-Detoxifiers (ICLR 2025 · similarity 0.86)
- Geometry of Decision Making in Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.