Text generated by OPUS-MT and T5 models with single-bit errors in the parameters

Description

The dataset contains text generated using T5 and OPUS-MT model with and with single-bit errors in the parameters of the LLM. The T5 LLM used the CNN Daily Mail dataset for summarization and OPUS-MT used the IWSLT2017 dataset for Chinese-to-English translation.

Folders:

t5_fp32: T5 model with a quantified version of FP32

t5_fp16: T5 model with a quantified version of FP16

opus_fp32: OPUS-MT model with a quantified version of FP32

opus_fp16: OPUS-MT model with a quantified version of FP16

Files:

{cnn/iwslt2017}_input_text.txt: Input text, that is, text to summarize (cnn and T5) or Chinese text to translate (iwslt2017 and OPUS-MT).  For each dataset in total there are number_input_texts.

{cnn/iwslt2017}_output_reference.txt: Example of result expected for CNN (T5) and IWSLT2017 (OPUS-MT). For each dataset in total there are number_input_texts.

{cnn/iwslt2017}_output_predict_fault_free: Example of predictions without single-bit errors. For each dataset in total there are number_input_texts.

{cnn/iwslt2017}_output_predict_single_fi_bit_100times: Example of predictions with 100 different single-bit error. In each dataset in total there are 100*number input texts.

Paper

Paper: Concurrent Linguistic Error Detection (CLED) for Large Language Models

Cite:

@misc{zhu2024concurrent,      title={Concurrent Linguistic Error Detection (CLED) for Large Language Models},       author={Jinhua Zhu and Javier Conde and Zhen Gao and Pedro Reviriego and Shanshan Liu and Fabrizio Lombardi},      year={2024},      eprint={2403.16393},      archivePrefix={arXiv},      primaryClass={cs.AI}}

Description

本数据集包含由T5与OPUS-MT模型生成的文本，涵盖大语言模型（Large Language Model，LLM）参数存在单比特错误与无单比特错误两类场景。其中，T5大语言模型以CNN每日邮报（CNN Daily Mail）数据集作为摘要生成任务的输入数据，OPUS-MT模型则以IWSLT2017数据集作为汉英机器翻译任务的输入数据。

Folders:

t5_fp32: 采用FP32量化版本的T5模型
t5_fp16: 采用FP16量化版本的T5模型
opus_fp32: 采用FP32量化版本的OPUS-MT模型
opus_fp16: 采用FP16量化版本的OPUS-MT模型

Files:

{cnn/iwslt2017}_input_text.txt: 输入文本文件，即针对CNN与T5场景的待摘要文本，或针对IWSLT2017与OPUS-MT场景的待翻译中文文本。每个数据集对应的输入文本总条数为number_input_texts。
{cnn/iwslt2017}_output_reference.txt: 对应CNN（T5）与IWSLT2017（OPUS-MT）任务的标准参考输出结果示例。每个数据集对应的参考输出总条数为number_input_texts。
{cnn/iwslt2017}_output_predict_fault_free: 无单比特错误场景下的模型预测结果示例。每个数据集对应的该类预测结果总条数为number_input_texts。
{cnn/iwslt2017}_output_predict_single_bit_100times: 存在100种不同单比特错误场景下的模型预测结果示例。每个数据集对应的该类预测结果总条数为100*number_input_texts。

Paper

相关论文：《面向大语言模型的并发语言错误检测（Concurrent Linguistic Error Detection, CLED）》
引用格式：

@misc{zhu2024concurrent, title={面向大语言模型的并发语言错误检测（Concurrent Linguistic Error Detection, CLED）}, author={Jinhua Zhu and Javier Conde and Zhen Gao and Pedro Reviriego and Shanshan Liu and Fabrizio Lombardi}, year={2024}, eprint={2403.16393}, archivePrefix={arXiv}, primaryClass={cs.AI}}