TCAB: Text Classification Attack Benchmark Dataset

TCAB is a large collection of successful adversarial attacks on state-of-the-art text classification models trained on multiple sentiment and abuse domain datasets. The dataset is broken up into 2 files: <em>train.csv</em>, and <em>val.csv</em>. The training set contains 1,448,751 instances (552,364 are "clean" unperturbed instances), while the validation set contains 482,914 instances (178,607 are "clean"). Each instance in both files have the following attributes: <strong>scenario</strong>: Domain, either <em>abuse</em> or <em>sentiment</em>. <strong>target_model_dataset</strong>: Dataset being attacked. <strong>target_model_train_dataset</strong>: Dataset the target model trained on. <strong>target_model</strong>: Type of victim model (e.g., <em>bert</em>, <em>roberta</em>, <em>xlnet</em>). <strong>attack_toolchain</strong>: Open-source attack toolchain, either TextAttack or OpenAttack. <strong>attack_name</strong>: Name of the attack method. <strong>original_text</strong>: Original input text. <strong>original_output</strong>: Prediction probabilities of the target model on the original text. <strong>ground_truth</strong>: Encoded label for the original task of the domain dataset. 1 and 0 means toxic and toxic for abuse datasets, respectively. 1 and 0 means positive and negative sentiment for sentiment datasets. If there is a neutral sentiment, then 2, 1, 0 means positive, neutral, and negative sentiment. <strong>status</strong>: Unperturbed example if "clean"; successful adversarial attack if "success". <strong>perturbed_text</strong>: Text after it has been perturbed by an attack. <strong>perturbed_output</strong>: Prediction probabilities of the target model on the perturbed text. <strong>attack_time</strong>: Time taken to execute the attack. <strong>num_queries</strong>: Number of queries performed while attacking. <strong>frac_words_changed</strong>: Fraction of words changed due to an attack. <strong>test_index</strong>: Index of each unique source example (original instance) (LEGACY - necessary for backwards compatibility). <strong>original_text_identifier</strong>: Index of each unique source example (original instance). <strong>unique_src_instance_identifier</strong>: Primary key to uniquely identify to every source instance; comprised of (<em>target_model_dataset</em>, <em>test_index</em>, <em>original_text_identifier</em>). <strong>pk</strong>: Primary key to uniquely identify every attack instance; comprised of (<em>attack_name</em>, <em>attack_toolchain</em>, <em>original_text_identifier</em>, <em>scenario</em>, <em>target_model</em>, <em>target_model_dataset</em>, <em>test_index).</em>TCAB是一个针对基于多类情感与辱骂领域数据集训练的当前顶尖文本分类模型的成功对抗攻击大型集合。该数据集分为两个文件：<em>train.csv</em>与<em>val.csv</em>。其中训练集包含1,448,751个样本（其中552,364个为“干净”的未扰动样本），验证集包含482,914个样本（其中178,607个为“干净”样本）。两个文件中的每个样本均包含以下属性：场景（scenario）：所属领域，分为辱骂（abuse）与情感（sentiment）两类；目标模型数据集（target_model_dataset）：受攻击的目标数据集；目标模型训练数据集（target_model_train_dataset）：目标模型所使用的训练数据集；受害模型类型（target_model）：例如bert（BERT）、roberta（RoBERTa）、xlnet（XLNet）；开源攻击工具链（attack_toolchain）：可选TextAttack或OpenAttack；攻击方法名称（attack_name）；原始输入文本（original_text）；目标模型在原始输入文本上的预测概率（original_output）；真实标签（ground_truth）：对应领域数据集原始任务的编码标签。对于辱骂数据集，1与0分别代表冒犯性与非冒犯性样本；对于情感数据集，1与0分别代表积极与消极情感。若存在中性情感，则2、1、0分别对应积极、中性与消极情感；样本状态（status）：若为“clean”则代表未扰动样本，若为“success”则代表成功的对抗攻击样本；扰动后文本（perturbed_text）：经攻击手段扰动后的文本；目标模型在扰动后文本上的预测概率（perturbed_output）；攻击耗时（attack_time）：执行攻击所花费的时间；查询次数（num_queries）：攻击过程中发起的查询总次数；词语修改占比（frac_words_changed）：攻击导致的被修改词语占总词语的比例；测试索引（test_index）：每个唯一源样本（原始实例）的索引，为遗留字段，用于保障向后兼容性；原始文本标识符（original_text_identifier）：每个唯一源样本（原始实例）的索引；唯一源实例标识符（unique_src_instance_identifier）：用于唯一标识每个源实例的主键，由（target_model_dataset、test_index、original_text_identifier）组合而成；主键（pk）：用于唯一标识每个攻击实例的主键，由（attack_name、attack_toolchain、original_text_identifier、scenario、target_model、target_model_dataset、test_index）组合而成。