FailureSensorIQ

# FailureSensorIQ Dataset

FailureSensorIQ is a Multi-Choice QA (MCQA) dataset that explores the relationships between sensors and failure modes for 10 industrial assets.

|[**Github**](https://github.com/IBM/FailureSensorIQ) | [**🏆Leaderboard**](https://huggingface.co/spaces/cc4718/FailureSensorIQ) | [**📖Paper**](https://arxiv.org/abs/2506.03278) |

## Dataset Summary
FailureSensorIQ is a Multi-Choice QA (MCQA) dataset that explores the relationships between sensors and failure modes for 10 industrial assets. By only leveraging the information found in ISO documents, we developed a data generation pipeline that creates questions in two types: (i) FailureMode2Sensor and (ii) Sensor2FailureMode. Additionally, we designed questions in a selection (select the relevant ones) vs. elimination (eliminate the irrelevant ones) format.

FailureSensorIQ dataset consists of 8,296 questions across 10 assets, with 2,667 single-true multi-choice questions and 5,629 multi-true multi-choice questions. The following is the list of assets with count within 2,667 single-true multi-choice questions:

| Asset | Count |
|-----------------------------|-------|
| Electric Motor | 234 |
| Steam Turbine | 171 |
| Aero Gas Turbine | 336 |
| Industrial Gas Turbine | 240 |
| Pump | 152 |
| Compressor | 220 |
| Reciprocating IC Engine | 336 |
| Electric Generator | 234 |
| Fan | 200 |
| Power Transformer | 544 |

Please find more statistics of the dataset and the dataset construction process from our [Github](https://github.com/IBM/FailureSensorIQ) and [Paper](https://arxiv.org/abs/2506.03278).

## Load the Dataset
To load 2,667 single-true multi-choice QA,

```python
from datasets import load_dataset

# Login using e.g. `huggingface-cli login` to access this dataset
ds = load_dataset("ibm-research/FailureSensorIQ", "single_true_multi_choice_qa")
```

To load 5,629 multi-true multi-choice QA,
```python
from datasets import load_dataset

# Login using e.g. `huggingface-cli login` to access this dataset
ds = load_dataset("ibm-research/FailureSensorIQ", "multi_true_multi_choice_qa")
```

## Leaderboard
We have benchmarked both open-source LLMs and frontier LLMs on FailureSensorIQ. Furthermore, we have conducted thoroughout Perturbation–Uncertainty–Complexity analysis for deeper insights of each LLMs. For the most updated leaderboard, please refer to our [Leaderboard](https://huggingface.co/spaces/cc4718/FailureSensorIQ). You can submit the evaluation there.

If you want to reproduce our results, please check out our [Github](https://github.com/IBM/FailureSensorIQ) for the evaluation scripts. We documented all the steps you should take to run the evaluation.

## Cite this Dataset
If you use our dataset in your paper, please cite our dataset by
```
@misc{constantinides2025failuresensoriqmultichoiceqadataset,
title={FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes},
author={Christodoulos Constantinides and Dhaval Patel and Shuxin Lin and Claudio Guerrero and Sunil Dagajirao Patil and Jayant Kalagnanam},
year={2025},
eprint={2506.03278},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2506.03278},
}
```

# FailureSensorIQ 数据集

FailureSensorIQ 是一款多选问答（Multi-Choice QA，MCQA）数据集，旨在探索10种工业资产的传感器与故障模式之间的关联。

|[**Github**](https://github.com/IBM/FailureSensorIQ) | [**🏆排行榜**](https://huggingface.co/spaces/cc4718/FailureSensorIQ) | [**📖论文**](https://arxiv.org/abs/2506.03278) |

## 数据集概述

本数据集为多选问答（Multi-Choice QA，MCQA）数据集，聚焦10种工业资产的传感器与故障模式关联挖掘。研究团队仅依托ISO标准文档中的信息搭建了数据生成流程，可生成两类问题：(i) 故障模式到传感器（FailureMode2Sensor）；(ii) 传感器到故障模式（Sensor2FailureMode）。此外，我们设计了两种题型：遴选型（选择相关项）与排除型（排除无关项）。

本数据集共包含10种工业资产的8296道问题，其中单正确多选问答问题2667道，多正确多选问答问题5629道。以下为2667道单正确多选问答问题对应的资产及题量统计：

| 资产名称 | 题量 |
|-----------------------------|-------|
| 电动机（Electric Motor） | 234 |
| 蒸汽轮机（Steam Turbine） | 171 |
| 航空燃气轮机（Aero Gas Turbine） | 336 |
| 工业燃气轮机（Industrial Gas Turbine） | 240 |
| 泵（Pump） | 152 |
| 压缩机（Compressor） | 220 |
| 往复式内燃机（Reciprocating IC Engine） | 336 |
| 发电机（Electric Generator） | 234 |
| 风机（Fan） | 200 |
| 电力变压器（Power Transformer） | 544 |

如需了解本数据集的更多统计信息与构建流程，请访问我们的[**Github**](https://github.com/IBM/FailureSensorIQ)与[**论文**](https://arxiv.org/abs/2506.03278)页面。

## 加载数据集

若需加载2667道单正确多选问答问题：
python
from datasets import load_dataset

# 例如执行 `huggingface-cli login` 进行登录以访问本数据集
ds = load_dataset("ibm-research/FailureSensorIQ", "single_true_multi_choice_qa")

若需加载5629道多正确多选问答问题：
python
from datasets import load_dataset

# 例如执行 `huggingface-cli login` 进行登录以访问本数据集
ds = load_dataset("ibm-research/FailureSensorIQ", "multi_true_multi_choice_qa")

## 排行榜

我们已在FailureSensorIQ数据集上对开源大语言模型（Large Language Model，LLM）与前沿大语言模型进行了基准测试。此外，我们还开展了全面的扰动-不确定性-复杂度分析，以深入剖析各类大语言模型的性能表现。如需查看最新排行榜，请访问我们的[**🏆排行榜**](https://huggingface.co/spaces/cc4718/FailureSensorIQ)页面，您也可在此提交模型评估结果。

若需复现我们的实验结果，请前往我们的[**Github**](https://github.com/IBM/FailureSensorIQ)页面获取评估脚本，其中已详细记录了运行评估所需的全部步骤。

## 引用本数据集

若您在论文中使用了本数据集，请按以下格式引用：

@misc{constantinides2025failuresensoriqmultichoiceqadataset,
title={FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes},
author={Christodoulos Constantinides and Dhaval Patel and Shuxin Lin and Claudio Guerrero and Sunil Dagajirao Patil and Jayant Kalagnanam},
year={2025},
eprint={2506.03278},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2506.03278},
}