Research map / Large language models / Language, safety and interpretability
Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViT
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACT-ViT, a Vision Transformer-style model that treats LLM activation tensors (layers × tokens × hidden) as images to detect hallucinations, trained jointly across multiple LLMs with per-LLM linear adapters.
Problem
Hallucination detection from LLM internals is limited by probing classifiers that use isolated layer-token pairs and are specific to a single LLM, which hinders cross-LLM training and transfer.
Approach
Treat full activation tensors as images with layer and token axes as spatial dimensions; pool them to a fixed size, map each LLM's hidden states into a shared space with a per-LLM linear adapter, and process them with a shared ViT backbone that can be trained on multiple LLMs and frozen for adaptation to new LLMs.
Key results
- ACT-ViT outperforms the best probing baseline Probe[*] on most of the 15 LLM-dataset combinations in-domain, e.g., 84.33 vs 79.88 AUC on Mis-7B HotpotQA.
- Zero-shot leave-one-dataset-out ACT-ViT beats Best-Probas in 13 of 15 cases, with a +37 AUC gain on IMDB with Mis-7B.
- Inference takes about 1e-5 seconds per instance and joint training on all 15 combinations takes under three hours on a single L40 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ACT-ViT instead of the best prior-work baseline | Hallucination detection · Mistral-7B / HotpotQA | AUC: +1.64 | Helped |
| Use ACT-ViT instead of the best prior-work baseline | Hallucination detection · Mistral-7B / Movies | AUC: +4.04 | Helped |
| Use ACT-ViT instead of the best prior-work baseline | Hallucination detection · Mistral-7B / HotpotQA with context | AUC: +3.43 | Helped |
| Use ACT-ViT instead of the best prior-work baseline | Hallucination detection · Mistral-7B / IMDB | AUC: -0.58 | Hurt |
| Train jointly across LLMs and datasets rather than separately per LLM-dataset pair (ablation) | ACT-ViT · 15 LLM-dataset combinations | AUC | Helped |
| Replace the ViT backbone with a flattened MLP (ablation) | ACT-ViT · In-domain hallucination detection | AUC | Hurt |
| Reduce pooled size to (L_p, N_p) = (4, 20) (ablation) | ACT-ViT(s) on Qwen2.5-7B · HotpotQA | AUC: 86% | Hurt |
| Train ACT-ViT on pooled activation tensors from multiple LLMs and datasets instead of a single pair (ablation) | ACT-ViT · In-domain hallucination detection | AUC: 12 out of 15 cases | Helped |
| Use only 10% of the target training data to fine-tune the adapter (ablation) | ACT-ViT with frozen ViT backbone · Mis-7B / HotpotQA | AUC: 10% | Helped |
| Fine-tune only a new LLM-specific adapter while freezing the backbone (ablation) | ACT-ViT · Unseen LLM and dataset | AUC: 13 out of 15 cases | Helped |
Surprising findings
- The best token position and layer for hallucination signals differ substantially across LLMs even on the same dataset.
- The ACT-MLP baseline is often outperformed by the single-position probe, showing the ViT inductive bias matters.
Limitations
- Max pooling of large activation tensors may discard informative signals.
- Evaluated on only three LLMs and five datasets, with hallucination defined as answer correctness.
Experimental setup
Datasets: HotpotQA HotpotQA with context Movies IMDB TriviaQA
Benchmarks: HotpotQA HotpotQA with context Movies IMDB TriviaQA
Metrics: AUC
Baselines: Logit/Probas-mean/min/max Probe[*] Token[n] probes ACT-MLP LOS-Net Best-Probas
Builds on: Vision Transformer linear probing universal truthfulness hypothesis
Closest papers by meaning
- Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations (CVPR 2026 · similarity 0.92)
- Scalable Token-Level Hallucination Detection in Large Language Models (NeurIPS 2026 · similarity 0.91)
- Steer LLM Latents for Hallucination Detection (ICML 2025 · similarity 0.91)
- Sample Transform Cost-Based Training-Free Hallucination Detector for Large Language Models (NeurIPS 2026 · similarity 0.91)
- HalLoc: Token-level Localization of Hallucinations for Vision Language Models (CVPR 2025 · similarity 0.90)
- One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.