Research map / Learning theory and algorithms / Combinatorial and search optimization
Detecting Egregious Responses in Neural Sequence-to-sequence Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sampling-based likelihood hits instead of requiring exact greedy output | Seq2seq dialogue models on Ubuntu · Ubuntu mal list | io-sample-avg hit rate: 0% -> 48.6% (last-h); 0% -> 49.2% (attention) | Helped |
| Constrain trigger inputs with a language-model likelihood requirement | gibbs-enum on Ubuntu mal targets · Ubuntu mal list | o-sample-min hit rate: 13.6% -> 9.1% (last-h); 16.7% -> 10.2% (attention) | Hurt |
| Use attention rather than last-h context | gibbs-enum on Ubuntu mal targets · Ubuntu mal list | o-sample-min hit rate: m13.6% -> m16.7% | Helped |
| Use attention rather than last-h context | gibbs-enum on Switchboard mal targets · Switchboard mal list | o-sample-min hit rate: m0% -> m0.1% | Mixed |
| Use attention rather than last-h context | gibbs-enum on OpenSubtitles mal targets · OpenSubtitles mal list | o-greedy hit rate: 3% -> 6.6% | Helped |
| Train on OpenSubtitles, which contains egregious dialogue | Seq2seq dialogue models · mal list, o-greedy hit rate | o-greedy hit rate: Ubuntu 0%; Switchboard 0%; OpenSubtitles 3% and 6.6% | Helped |
| Allow a looser likelihood threshold with k=2 rather than k=1 (ablation) | gibbs-enum on Ubuntu mal targets, last-h · Ubuntu mal list | io-sample-avg hit rate: a48.6% -> a73.4% | Helped |
| Optimize a continuous relaxation and project the result to one-hot inputs (ablation) | Ubuntu attention model, mal target list · Ubuntu mal list | greedy target hit rate: 1.7% -> 0% | Hurt |
| Add an input language-model score constraint | gibbs-enum trigger inputs · Ubuntu mal list | trigger-input grammaticality | Helped |
| Increase random initializations and candidate enumeration count G (ablation) | gibbs-enum on Ubuntu normal targets, last-h · Ubuntu normal list | o-greedy hit rate: saturates at around 60% | Helped |
| Search arbitrary random target sequences | gibbs-enum on Ubuntu, Switchboard, and OpenSubtitles · Random target lists | hit rate: 0% | Hurt |
Closest papers by meaning
- Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples (AAAI 2020 · similarity 0.87)
- Minimum Divergence vs. Maximum Margin: an Empirical Comparison on Seq2Seq Models (ICLR 2019 · similarity 0.85)
- Guiding Attention in Sequence-to-Sequence Models for Dialogue Act Prediction (AAAI 2020 · similarity 0.85)
- Eliciting Language Model Behaviors with Investigator Agents (ICML 2025 · similarity 0.84)
- SeqGPT: An Out-of-the-Box Large Language Model for Open Domain Sequence Understanding (AAAI 2024 · similarity 0.83)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.