openGPT-X/truthfulqax

---
language:
- de
- fr
- es
- it
- pt
- bg
- cs
- da
- el
- et
- fi
- hu
- lt
- lv
- nl
- pl
- ro
- sk
- sl
- sv
task_categories:
- text-generation
pretty_name: EU-20-TruthfulQA
size_categories:
- 10K<n<100K
---

### Citation Information

If you find benchmarks useful in your research, please consider citing the test and also the [TruthfulQA](https://aclanthology.org/2022.acl-long.229) dataset it draws from:
```
@misc{thellmann2024crosslingual,
title={Towards Cross-Lingual LLM Evaluation for European Languages},
author={Klaudia Thellmann and Bernhard Stadler and Michael Fromm and Jasper Schulze Buschhoff and Alex Jude and Fabio Barth and Johannes Leveling and Nicolas Flores-Herr and Joachim Köhler and René Jäkel and Mehdi Ali},
year={2024},
eprint={2410.08928},
archivePrefix={arXiv},
primaryClass={cs.CL}

# TruthfulQA
@inproceedings{lin-etal-2022-truthfulqa,
title = "{T}ruthful{QA}: Measuring How Models Mimic Human Falsehoods",
author = "Lin, Stephanie and
Hilton, Jacob and
Evans, Owain",
editor = "Muresan, Smaranda and
Nakov, Preslav and
Villavicencio, Aline",
booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.acl-long.229",
doi = "10.18653/v1/2022.acl-long.229",
pages = "3214--3252",
abstract = "We propose a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. We crafted questions that some humans would answer falsely due to a false belief or misconception. To perform well, models must avoid generating false answers learned from imitating human texts. We tested GPT-3, GPT-Neo/J, GPT-2 and a T5-based model. The best model was truthful on 58{\%} of questions, while human performance was 94{\%}. Models generated many false answers that mimic popular misconceptions and have the potential to deceive humans. The largest models were generally the least truthful. This contrasts with other NLP tasks, where performance improves with model size. However, this result is expected if false answers are learned from the training distribution. We suggest that scaling up models alone is less promising for improving truthfulness than fine-tuning using training objectives other than imitation of text from the web.",
}
}
```