TCAB: Text Classification Attack Benchmark Dataset

TCAB is a large collection of successful adversarial attacks on state-of-the-art text classification models trained on multiple sentiment and abuse domain datasets. The dataset is broken up into 2 files: <em>train.csv and</em> <em>val.csv</em>. The training set contains 1,448,751 instances (552,364 are "clean" unperturbed instances) and the validation set contains 482,914 instances (178,607 are "clean"). Each instance contains the following attributes: <strong>scenario</strong>: Domain, either <em>abuse</em> or <em>sentiment</em>. <strong>target_model_dataset</strong>: Dataset being attacked. <strong>target_model_train_dataset</strong>: Dataset the target model trained on. <strong>target_model</strong>: Type of victim model (e.g., <em>bert</em>, <em>roberta</em>, <em>xlnet</em>). <strong>attack_toolchain</strong>: Open-source attack toolchain, either TextAttack or OpenAttack. <strong>attack_name</strong>: Name of the attack method. <strong>original_text</strong>: Original input text. <strong>original_output</strong>: Prediction probabilities of the target model on the original text. <strong>ground_truth</strong>: Encoded label for the original task of the domain dataset. 1 and 0 means toxic and toxic for abuse datasets, respectively. 1 and 0 means positive and negative sentiment for sentiment datasets. If there is a neutral sentiment, then 2, 1, 0 means positive, neutral, and negative sentiment. <strong>status</strong>: Unperturbed example if "clean"; successful adversarial attack if "success". <strong>perturbed_text</strong>: Text after it has been perturbed by an attack. <strong>perturbed_output</strong>: Prediction probabilities of the target model on the perturbed text. <strong>attack_time</strong>: Time taken to execute the attack. <strong>num_queries</strong>: Number of queries performed while attacking. <strong>frac_words_changed</strong>: Fraction of words changed due to an attack. <strong>test_index</strong>: Index of each unique source example (original instance) (LEGACY - necessary for backwards compatibility). <strong>original_text_identifier</strong>: Index of each unique source example (original instance). <strong>unique_src_instance_identifier</strong>: Primary key to uniquely identify to every source instance; comprised of (<em>target_model_dataset</em>, <em>test_index</em>, <em>original_text_identifier</em>). <strong>pk</strong>: Primary key to uniquely identify every attack instance; comprised of (<em>attack_name</em>, <em>attack_toolchain</em>, <em>original_text_identifier</em>, <em>scenario</em>, <em>target_model</em>, <em>target_model_dataset</em>, <em>test_index).</em>