Emanalejawi/OffTopicEval
收藏Hugging Face2025-12-09 更新2025-12-20 收录
下载链接：
https://hf-mirror.com/datasets/Emanalejawi/OffTopicEval
下载链接
链接失效反馈官方服务：
资源简介：
---
dataset_info:
- config_name: Chinese
features:
- name: in_domain
dtype: string
- name: subject
dtype: string
- name: question
dtype: string
- name: choices
list: string
- name: answer
dtype: string
- name: idx
dtype: int64
- name: origin_question
dtype: string
- name: attack_prompt
dtype: string
- name: sample_id
dtype: string
- name: translated_question
dtype: string
- name: id
dtype: string
- name: language
dtype: string
- name: split_type
dtype: string
splits:
- name: in
num_bytes: 458794
num_examples: 1050
- name: out
num_bytes: 125318177
num_examples: 70371
download_size: 52752257
dataset_size: 125776971
- config_name: English
features:
- name: in_domain
dtype: string
- name: subject
dtype: string
- name: question
dtype: string
- name: choices
list: string
- name: answer
dtype: string
- name: idx
dtype: int64
- name: origin_question
dtype: string
- name: attack_prompt
dtype: string
- name: sample_id
dtype: string
- name: translated_question
dtype: string
- name: id
dtype: string
- name: language
dtype: string
- name: split_type
dtype: string
splits:
- name: in
num_bytes: 302658
num_examples: 1050
- name: out
num_bytes: 294757930
num_examples: 70371
download_size: 74337106
dataset_size: 295060588
- config_name: Hindi
features:
- name: in_domain
dtype: string
- name: subject
dtype: string
- name: question
dtype: string
- name: choices
list: string
- name: answer
dtype: string
- name: idx
dtype: int64
- name: origin_question
dtype: string
- name: attack_prompt
dtype: string
- name: sample_id
dtype: string
- name: translated_question
dtype: string
- name: id
dtype: string
- name: language
dtype: string
- name: split_type
dtype: string
splits:
- name: in
num_bytes: 767230
num_examples: 1050
- name: out
num_bytes: 371004847
num_examples: 70371
download_size: 118257249
dataset_size: 371772077
configs:
- config_name: Chinese
data_files:
- split: in
path: Chinese/in-*
- split: out
path: Chinese/out-*
- config_name: English
data_files:
- split: in
path: English/in-*
- split: out
path: English/out-*
- config_name: Hindi
data_files:
- split: in
path: Hindi/in-*
- split: out
path: Hindi/out-*
task_categories:
- text-classification
language:
- en
- zh
- hi
tags:
- llm-safety
- operational-safety
- multilingual
- benchmark
---
# OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!
Paper: [https://huggingface.co/papers/2509.26495](https://huggingface.co/papers/2509.26495)
Code: [https://github.com/declare-lab/OffTopicEval](https://github.com/declare-lab/OffTopicEval)
**Note**: We release OffTopicEval, a multilingual evaluation suite for measuring operational safety of large language models (LLMs). The benchmark includes in-domain (ID), direct out-of-domain (OOD), and adaptive OOD queries, across English, Chinese, and Hindi.
If your work involves adaptive OOD analysis, please ensure you download the full dataset version, as it includes adversarially transformed queries generated using Llama-70B.
For multilingual evaluation, the dataset integrates translated data of Chinese and Hindi.
The dataset is large-scale (220K+ queries). We recommend users access it via Hugging Face Datasets API or the full release on GitHub for efficiency.
Thank you for your support of OffTopicEval — we hope it is useful for your research on safe and reliable LLM deployment.
## 📊 Dataset Description
OffTopicEval is the first multilingual benchmark for operational safety of LLMs, focusing on whether purpose-specific AI agents can:
Appropriately accept in-domain queries, and
Reliably refuse out-of-domain queries (both direct and adversarially adapted).
## 🔹 Key Features:
21 purpose-specific agents:
bankhelper,
bookingbot,
carecompanion,
careercoach,
enrollbot,
hrhelper,
linguabuddy,
loadguide,
localguide,
loyaltybuddy,
medischeduler,
mindease,
onboardhelper,
orderguide,
payhelper,
policybuddy,
recruitbot,
supportgenie,
travelcompanion,
tripplanner,
workplaceassistant
3,150 ID queries, 10,053 direct OOD queries, and 211,113 adaptive OOD queries.
Multilingual: English, Chinese, Hindi.
Evaluation Metrics:
AR<sub>ID</sub> – Acceptance rate for ID queries.
RR<sub>OOD</sub><sup>D</sup> – Refusal rate for direct OOD queries.
RR<sub>OOD</sub><sup>A</sup> – Refusal rate for adaptive OOD queries.
OS – Operational safety score (harmonic mean of AR<sub>ID</sub> and RR<sub>OOD</sub>).
## Citation
If you find our work useful, please cite:
```bibtex
@article{lei2025offtopiceval,
title={OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!},
author={Lei, Jingdi and Gumma, Varun and Bhardwaj, Rishabh and Lim, Seok Min and Li, Chuan and Zadeh, Amir and Poria, Soujanya},
year={2025},
journal={arXiv preprint arXiv:2509.26495},
url={https://arxiv.org/abs/2509.26495}
}
```
提供机构：
Emanalejawi
