FailSafeQA

Benchmark data introduced in the paper: \
**Expect the Unexpected: FailSafeQA Long Context for Finance** ([https://arxiv.org/abs/2502.06329](https://arxiv.org/abs/2502.06329))

Dataset count: 220

<img src="FailSafeQA.png" width="700" title="FailSafeQA">

```json
{
"idx": int,
"tokens": int,
"context": string,
"ocr_context": string,
"answer": string,
"query": string,
"incomplete_query": string,
"out-of-domain_query": string,
"error_query": string,
"out-of-scope_query": string,
"citations": string,
"citations_tokens": int
}
```

Tasks variety:
<img src="verb_dobj_base_new.png" width="400" title="FailSafeQA">
Root verbs and their direct objects from the first sentence of each normalized query, the top 20 verbs and their top five direct object.
Tasks types:
- 83.0% question answering (QA)
- 17.0% involving text generation (TG)

To cite FailSafeQA, please use:
```
@misc{kiran2024failsafeqa,
title={Expect the Unexpected: FailSafeQA Long Context for Finance},
author={Kiran Kamble and Melisa Russak and Dmytro Mozolevskyi and Muayad Ali and Mateusz Russak and Waseem AlShikh},
year={2024},
eprint={2502.06329},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

本论文提出的基准数据集：**《意料之外：面向金融领域的长上下文FailSafeQA数据集》**（论文链接：https://arxiv.org/abs/2502.06329）

数据集规模：220条
<img src="FailSafeQA.png" width="700" title="FailSafeQA">

单条数据的字段结构如下：
json
{
"索引（idx）": 整数,
"Token数量（tokens）": 整数,
"上下文（context）": 字符串,
"OCR上下文（ocr_context）": 字符串,
"答案（answer）": 字符串,
"查询（query）": 字符串,
"不完整查询（incomplete_query）": 字符串,
"域外查询（out-of-domain_query）": 字符串,
"错误查询（error_query）": 字符串,
"越域查询（out-of-scope_query）": 字符串,
"引用文献（citations）": 字符串,
"引用文献Token数（citations_tokens）": 整数
}

任务多样性：<img src="verb_dobj_base_new.png" width="400" title="FailSafeQA"> 该统计基于每条标准化查询的首句提取核心动词及其直接宾语，展示了排名前20的动词及其对应的前5个直接宾语。

任务类型：
- 83.0% 为问答（QA）任务
- 17.0% 涉及文本生成（TG）任务

若需引用FailSafeQA数据集，请使用以下格式：

@misc{kiran2024failsafeqa,
title={Expect the Unexpected: FailSafeQA Long Context for Finance},
author={Kiran Kamble and Melisa Russak and Dmytro Mozolevskyi and Muayad Ali and Mateusz Russak and Waseem AlShikh},
year={2024},
eprint={2502.06329},
archivePrefix={arXiv},
primaryClass={cs.CL}
}