Research map / Large language models / Language, safety and interpretability
Detecting Errors in AI-Generated Annotations: When and Why Semantic Neighbors Help
Na Di, Ling Li, Zhe Tang, Hao Cheng, Jinlong Pang, Jiaheng Wei, Zhaowei Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add kNN semantic-neighbor references to judge (SAGE vs Direct) | LLM judge · HaluEval (LLaMA-3.1) | AUROC (%): 66.65 -> 70.47 | Improved |
| add kNN semantic-neighbor references to judge (SAGE vs Direct) | LLM judge · HaluEval (Ministral) | AUROC (%): 57.61 -> 67.26 | Improved |
| add kNN references on TruthfulQA | LLM judge · TruthfulQA (LLaMA-3.1) | AUROC (%): 58.57 -> 58.15 | Hurt |
| add kNN references on MMLU | LLM judge · MMLU (Qwen3) | AUROC (%): 67.85 -> 68.99 (Random 70.84) | Hurt |
| replace random references with kNN-retrieved references | LLM judge · AG News | AUROC (%): 54.33 -> 62.44 | Improved |
| add kNN references to VLM captioning | VLM judge · Flickr30k | AUROC (%): +1.4 to +2.1 points over Direct | Improved |
| add kNN references on COCO captioning | VLM judge · COCO (Qwen3-VL) | AUROC (%): 55.80 -> 55.66 | No clear effect |
| add kNN references on CIFAR-10 | VLM judge · CIFAR-10 (InternVL3.5) | AUROC (%): 72.03 -> 92.99 | Improved |
| add kNN references on CIFAR-10 SAIL-VL2 | VLM judge · CIFAR-10 (SAIL-VL2) | AUROC (%): 74.16 -> 66.93 | Hurt |
| use unified external encoder for retrieval instead of self-embeddings (ablation) | SAGE · ImageNet-1k (Qwen3-VL) | AUROC (%): 65.71 -> 75.29 | Improved |
| use full-input retrieval instead of question-only embeddings (ablation) | SAGE · TruthfulQA (LLaMA-3.1) | AUROC (%): 58.15 -> 60.28 | Improved |
| keep judge in the pipeline versus bypassing it (SAGE vs DS2) (ablation) | SAGE · TruthfulQA (Qwen3) | AUROC (%): 67.10 -> 72.61 | Improved |
Closest papers by meaning
- LEMoN: Label Error Detection using Multimodal Neighbors (ICML 2025 · similarity 0.86)
- ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking (NeurIPS 2025 · similarity 0.85)
- Idiosyncrasies in Large Language Models (ICML 2025 · similarity 0.85)
- Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves? (CVPR 2025 · similarity 0.85)
- TLDR: Token-Level Detective Reward Model for Large Vision Language Models (ICLR 2025 · similarity 0.85)
- Learning from others' mistakes: Finetuning machine translation models with span-level error annotations (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
