defenseunicorns/LFAI_RAG_qa_v1

---
language:
- en
license: apache-2.0
configs:
- config_name: LFAI_RAG_qa_v1
data_files:
- split: eval
path: LFAI_RAG_qa_v1.json
default: true
---

# LFAI_RAG_qa_v1

This dataset aims to be the basis for RAG-focused question and answer evaluations for [LeapfrogAI](https://github.com/defenseunicorns/leapfrogai)🐸.

## Dataset Details

LFAI_RAG_qa_v1 contains 36 question/answer/context entries that are intended to be used for LLM-as-a-judge enabled RAG Evaluations.

Example:
```
{
"input": "What requirement must be met to run VPI PVA algorithms in a Docker container?",
"actual_output": null,
"expected_output": "To run VPI PVA algorithms in a Docker container, the same VPI version must be installed on the Docker host.",
"context": [
"2.6.\nCompute\nStack\nThe\nfollowing\nDeep\nLearning-related\nissues\nare\nnoted\nin\nthis\nrelease.\nIssue\nDescription\n4564075\nTo\nrun\nVPI\nPVA\nalgorithms\nin\na\ndocker\ncontainer,\nthe\nsame\nVPI\nversion\nhas\nto\nbe\ninstalled\non \nthe\ndocker\nhost.\n2.7.\nDeepstream\nIssue\nDescription\n4325898\nThe\npipeline\ngets\nstuck\nfor\nmulti\u0000lesrc\nwhen\nusing\nnvv4l2decoder.\nDS\ndevelopers\nuse \nthe\npipeline\nto\nrun\ndecode\nand\ninfer\njpeg\nimages.\nNVIDIA\nJetson\nLinux\nRelease\nNotes\nRN_10698-r36.3\n|\n11"
],
"source_file": "documents/Jetson_Linux_Release_Notes_r36.3.pdf"
}
```

### Dataset Sources

Data was generated from the following sources:

```
https://www.humanesociety.org/sites/default/files/docs/HSUS_ACFS-2023.pdf
https://www.whitehouse.gov/wp-content/uploads/2024/04/Global-Health-Security-Strategy-2024-1.pdf
https://www.armed-services.senate.gov/imo/media/doc/fy24_ndaa_conference_executive_summary1.pdf
https://dodcio.defense.gov/Portals/0/Documents/Library/(U)%202024-01-02%20DoD%20Cybersecurity%20Reciprocity%20Playbook.pdf
https://assets.ctfassets.net/oggad6svuzkv/2pIQQWQXPpxiKjjmhfpyWf/eb17b3f3c9c21f7abb05e68c7b1f3fcd/2023_annual_report.pdf
https://www.toyota.com/content/dam/toyota/brochures/pdf/2024/T-MMS-24Corolla.pdf
https://docs.nvidia.com/jetson/archives/r36.3/ReleaseNotes/Jetson_Linux_Release_Notes_r36.3.pdf
https://arxiv.org/pdf/2406.05370.pdf
```

The documents themselves can be found in [document_context.zip](https://huggingface.co/datasets/jalling/LFAI_RAG_qa_v1/raw/main/document_context.zip).

## Uses

This dataset is ready to be used for LLM-as-a-judge evaluations, formatted specifically for compatibility with [DeepEval](https://github.com/confident-ai/deepeval).

## Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

This dataset follows the format for Test Case [Goldens](https://docs.confident-ai.com/docs/confident-ai-manage-datasets#what-is-a-golden) in DeepEval.

Each entry in this dataset contains the following fields:
- `input`, the question to be prompted to your LLM
- `expected_output`, the ground truth answer to the prompted question
- `context`, the ground truth source in documentation that contains or informs the ground truth answer

## Dataset Creation

This dataset was generated from the source documentation using DeepEval's [Synthesizer](https://docs.confident-ai.com/docs/evaluation-datasets-synthetic-data).

The dataset was then refined by:
- Removing entries with poorly formatted or too simplistic questions
- Removing entries with question/answer pairs that did not make sense in context
- Modifying questions to reduce verbosity and increase factual accuracy

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

This dataset was generated using GPT-4o, and therefore carries along the bias of the model as well as the human annotator who refined it.

The dataset was created with the intention of using source documentation that is unlikely to be in the training data of any current models, but this will likely change within the coming months as new models are released.

## Dataset Card Authors
The Leapfrogai🐸 team at [Defense Unicorns](https://www.defenseunicorns.com/)🦄

## Dataset Card Contact
- [e-mail]

---
语言：
- 英语
许可证：Apache 2.0
配置项：
- 配置名称：LFAI_RAG_qa_v1
数据文件：
- 拆分集：评估集（eval）
路径：LFAI_RAG_qa_v1.json
默认配置：是
---

# LFAI_RAG_qa_v1 数据集

本数据集旨在作为面向[LeapfrogAI](https://github.com/defenseunicorns/leapfrogai)🐸的检索增强生成（Retrieval-Augmented Generation, RAG）相关问答评估的基础。

## 数据集详情

LFAI_RAG_qa_v1 包含36条问答与上下文条目，适用于启用大语言模型（Large Language Model, LLM）作为评判器的RAG评估任务。

示例：
json
{
"input": "在Docker容器中运行VPI PVA算法需满足何种要求？",
"actual_output": null,
"expected_output": "若要在Docker容器中运行VPI PVA算法，Docker宿主机上需安装与容器相同版本的VPI。",
"context": [
"2.6 计算栈：本版本中记录了以下与深度学习相关的问题。
问题编号：4564075，问题描述：若要在Docker容器中运行VPI PVA算法，Docker宿主机上需安装与容器相同版本的VPI。
2.7 DeepStream：
问题编号：4325898，问题描述：使用nvv4l2decoder时，多源输入管道会陷入停滞。DS开发人员会使用该管道完成JPEG图像的解码与推理任务。
NVIDIA Jetson Linux 发布说明 RN_10698-r36.3 | 11"
],
"source_file": "documents/Jetson_Linux_Release_Notes_r36.3.pdf"
}

### 数据集来源

本数据集的数据源自以下公开文档：

https://www.humanesociety.org/sites/default/files/docs/HSUS_ACFS-2023.pdf
https://www.whitehouse.gov/wp-content/uploads/2024/04/Global-Health-Security-Strategy-2024-1.pdf
https://www.armed-services.senate.gov/imo/media/doc/fy24_ndaa_conference_executive_summary1.pdf
https://dodcio.defense.gov/Portals/0/Documents/Library/(U)%202024-01-02%20DoD%20Cybersecurity%20Reciprocity%20Playbook.pdf
https://assets.ctfassets.net/oggad6svuzkv/2pIQQWQXPpxiKjjmhfpyWf/eb17b3f3c9c21f7abb05e68c7b1f3fcd/2023_annual_report.pdf
https://www.toyota.com/content/dam/toyota/brochures/pdf/2024/T-MMS-24Corolla.pdf
https://docs.nvidia.com/jetson/archives/r36.3/ReleaseNotes/Jetson_Linux_Release_Notes_r36.3.pdf
https://arxiv.org/pdf/2406.05370.pdf

所有源文档均可通过[document_context.zip](https://huggingface.co/datasets/jalling/LFAI_RAG_qa_v1/raw/main/document_context.zip)获取。

## 用途

本数据集已就绪，可用于以大语言模型作为评判器的RAG评估任务，其格式专为适配[DeepEval](https://github.com/confident-ai/deepeval)而设计。

## 数据集结构

本部分将对数据集字段进行说明，并补充数据集拆分规则、数据点间关联关系等结构相关信息。

本数据集遵循DeepEval中测试用例金标准样本（Goldens）的格式规范。
数据集中的每条条目均包含以下字段：
- `input`：用于向大语言模型发起提问的提示词
- `expected_output`：对应提问的标准答案
- `context`：包含或支撑标准答案的官方文档源上下文

## 数据集构建

本数据集通过DeepEval的[Synthesizer模块](https://docs.confident-ai.com/docs/evaluation-datasets-synthetic-data)从源文档中生成。
随后通过以下步骤对数据集进行优化：
- 移除格式错误或过于简单的提问条目
- 移除问答对与上下文不符的条目
- 精简提问表述，提升事实准确性

## 偏差、风险与局限性

本部分旨在说明数据集在技术与社会技术层面的局限性。
本数据集通过GPT-4o生成，因此不可避免地带有该模型以及参与优化的人工标注者所携带的偏差。
本数据集的构建初衷是选用当前主流大语言模型训练数据中不太可能包含的源文档，但随着新模型的持续发布，这一情况可能在未来数月内发生变化。

## 数据集卡片作者

来自[Defense Unicorns](https://www.defenseunicorns.com/)🦄的LeapfrogAI🐸团队

## 数据集卡片联系方式
- [e-mail]