Research map / Vision-language and multimodal / Human and embodied modeling
Textual Training for the Hassle-Free Removal of Unwanted Visual Data: Case Studies on OOD and Hateful Image Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HFTT, which trains trainable text-side embeddings in CLIP's joint space using only synthetic text (word-template prompts) to detect OOD and hateful images without any images.
Problem
Detecting unwanted visual content (OOD images, hateful images) normally requires in-distribution or outlier images and costly vision-language training; the paper asks whether textual data alone can train such detectors.
Approach
A theoretical argument that a classifier trained on one modality of a well-aligned bimodal embedding space transfers to the other, combined with a modified focal-weighted loss that treats all synthetic text as out-distribution and a rule-based synthesis of text from a 370k English word set and prompt templates; only N trainable embeddings in CLIP's output space are learned with the backbone frozen
Key results
- On ImageNet-1K OOD, HFTT reaches 91.76 average AUROC and 33.33 average FPR95, outperforming in-distribution-image baselines such as NPOS (91.22 AUROC, 37.94 FPR).
- On the Hate dataset, HFTT achieves 99.06 average AUROC and 1.83 FPR95, versus 98.82 and 4.49 for NegLabel.
- Updating the trainable embeddings takes under 2 minutes on a single V100 GPU for the ~370k-word corpus with CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HFTT instead of NPOS | CLIP ViT-B/16 · ImageNet-1K OOD average | FPR: 37.94 -> 33.33 | Helped |
| use HFTT instead of NPOS | CLIP ViT-B/16 · ImageNet-1K OOD average | AUROC: 91.22 -> 91.76 | Helped |
| use HFTT instead of NegLabel | CLIP ViT-B/16 · Hate innocuous-data average | FPR: 4.49 -> 1.83 | Helped |
| use HFTT instead of NegLabel | CLIP ViT-B/16 · Hate innocuous-data average | AUROC: 98.82 -> 99.06 | Helped |
| use HFTT instead of MCM | CLIP ViT-B/16 · ImageNet-C | AUROC: 89.06 -> 92.81 | Helped |
| add medical-domain descriptions to HFTT (ablation) | HFTT with CLIP · PVQA | AUROC: 93.60 -> 97.05 | Helped |
| add medical-domain descriptions to HFTT (ablation) | HFTT with CLIP · PCAM | AUROC: 96.94 -> 98.35 | Helped |
| add medical-domain corpus engineering to HFTT with descriptions (ablation) | HFTT with CLIP · PVQA | AUROC: 97.05 -> 98.69 | Helped |
| add medical-domain corpus engineering to HFTT with descriptions (ablation) | HFTT with CLIP · PCAM | AUROC: 98.35 -> 98.73 | Helped |
| use focal loss with gamma=1 instead of gamma=0 (ablation) | HFTT · ImageNet OOD average | AUROC: 88.19 -> 88.30 | Helped |
| use temperature 1.0 instead of 0.01 (ablation) | HFTT · ImageNet OOD average | AUROC: 88.30 -> 46.34 | Hurt |
| increase trainable embeddings from 10 to 2000 (ablation) | HFTT · ImageNet OOD average | AUROC: 88.30 -> 88.32 | No clear effect |
Surprising findings
- Trained embeddings land in OOD regions of CLIP's joint space even though no visual OOD data was used in training.
- HFTT with only 5 trainable embeddings performs nearly as well as with 2000 on ImageNet OOD.
Limitations
- Capability is bounded by the representative capacity of the pre-trained VLM.
- Requires text-image alignment, so the text encoder must match the vision encoder (e.g., CLIP with CLIP).
Experimental setup
Datasets: ImageNet-1K iNaturalist SUN Places Textures NINCO Hate dataset ImageNet-C ISIC-18 PathVQA PatchCamelyon english-words word set
Benchmarks: ImageNet-1K OOD (iNaturalist, SUN, Places, Textures, NINCO) Hate dataset (innocuous OOD) ImageNet-C low-quality detection medical OOD (PathVQA, PCAM)
Metrics: AUROC FPR95
Baselines: Mahalanobis MSP KNN NPOS Energy ZOC MaxLogit MCM CLIPN NegLabel
Builds on: CLIP focal loss MCM NegLabel
Closest papers by meaning
- Language-Assisted Feature Transformation for Anomaly Detection (ICLR 2025 · similarity 0.84)
- ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training (NeurIPS 2025 · similarity 0.83)
- Generating Enhanced Negatives for Training Language-Based Object Detectors (CVPR 2024 · similarity 0.83)
- MANTA: A Large-Scale Multi-View and Visual-Text Anomaly Detection Dataset for Tiny Objects (CVPR 2025 · similarity 0.83)
- SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation (ICLR 2025 · similarity 0.83)
- T-MARS: Improving Visual Representations by Circumventing Text Feature Learning (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.