jon-tow/okapi_truthfulqa

TruthfulQA是一个用于衡量语言模型在生成答案时是否真实的基准测试。该基准测试包含817个问题，涵盖38个类别，如健康、法律、金融和政治等。问题设计使得一些人类可能会因为错误信念或误解而给出错误答案。为了表现良好，模型必须避免生成从模仿人类文本中学到的错误答案。数据集由多位研究者共同策划，并采用CC BY NC 4.0许可证，仅允许非商业用途。

TruthfulQA is a benchmark for evaluating whether language models produce truthful responses during answer generation. This benchmark consists of 817 questions across 38 categories including health, law, finance, politics, and more. The questions are intentionally designed such that humans may incorrectly answer them due to false beliefs or misperceptions. To achieve strong performance on this benchmark, models must avoid generating incorrect answers learned through mimicking human-written text. This dataset was co-curated by multiple researchers and is released under the CC BY NC 4.0 license, which only allows non-commercial usage.

原始信息汇总

okapi_truthfulqa

数据集概述

okapi_truthfulqa 是一个多语言版本的 TruthfulQA 数据集。TruthfulQA 是一个用于衡量语言模型在生成问题答案时是否真实的基准测试。该基准包含 817 个问题，涵盖 38 个类别，包括健康、法律、金融和政治等领域。这些问题设计得使一些人类会因错误信念或误解而给出错误答案。为了表现良好，模型必须避免生成从模仿人类文本中学习到的错误答案。

数据集详情

数据集描述

• 创建者： Dac Lai, Viet 和 Van Nguyen, Chien 和 Ngo, Nghia Trung 和 Nguyen, Thuat 和 Dernoncourt, Franck 和 Rossi, Ryan A 和 Nguyen, Thien Huu
• 许可证： CC BY NC 4.0（仅允许非商业使用）

数据集来源

• 仓库： http://nlp.uoregon.edu/download/okapi-eval/datasets/
• 论文： Okapi (Lai et al., 2023)

引用

bibtex @article{dac2023okapi, title={Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback}, author={Dac Lai, Viet and Van Nguyen, Chien and Ngo, Nghia Trung and Nguyen, Thuat and Dernoncourt, Franck and Rossi, Ryan A and Nguyen, Thien Huu}, journal={arXiv e-prints}, pages={arXiv--2307}, year={2023} }

bibtex @misc{lin2021truthfulqa, title={TruthfulQA: Measuring How Models Mimic Human Falsehoods}, author={Stephanie Lin and Jacob Hilton and Owain Evans}, year={2021}, eprint={2109.07958}, archivePrefix={arXiv}, primaryClass={cs.CL} }

搜集汇总

背景与挑战

背景概述

okapi_truthfulqa是多语言版本的TruthfulQA基准测试数据集，包含817个涵盖38个类别的问题，用于评估语言模型回答问题的真实性。该数据集设计特殊问题以测试模型避免生成虚假答案的能力，采用CC BY NC 4.0许可证仅限非商业用途。