Dataset for: Metamorphic Testing of Large Language Models for Natural Language Processing (ICSME'25)

# Dataset for: Metamorphic Testing of Large Language Models for Natural Language Processing (ICSME'25) ## Overview This repository contains the data associated with the paper [Metamorphic Testing of Large Language Models using Natural Language Processing](https://valerio-terragni.github.io/assets/pdf/cho-icsme-2025.pdf), accepted at ICSME 2025. The associated tool, LLMorph, can be found at: https://github.com/steven-b-cho/llmorph Authors: Steven Cho, Stefano Ruberto, Valerio Terragni If you use this dataset, please cite: ```@inproceedings{cho2025metamorphic, author = {Cho, Steven and Ruberto, Stefano and Terragni, Valerio}, title = {Metamorphic Testing of Large Language Models for Natural Language Processing}, booktitle = {Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME)}, year = {2025}, publisher = {IEEE}}``` ## Description All data is given as JSON files. ```mt4llm-data/├── rq1-failure-rate/results/├── rq2-ground-truth-comparison/│ ├── source-inputs-with-labels/│ ├── source-output-correctness/│ └── ground_truth_comparison_summary.json├── rq3-manual-validation/manually_labelled_violations.json├── rqextra-flakiness/│ ├── results/│ ├── fail_rates.json│ ├── fail_rates_summarised.json│ └── manual_vs_flakiness_comparison.json├── summary/│ ├── data_summary_by_category.json│ └── data_summary_by_cid.json└── README.md``` ### RQ1 `rq1/results` contains the raw output of the data for testing relations on tasks on LLMs. The properties are: - `llm_name`, `task_name`, `relation_name`: The used LLM, task, and MR, respectively.- `data`: The list of data instances. - `id`: The nth instance in this file. - `source_input`, `source_output`, `followup_input`, `followup_output`: The source and follow-up inputs and outputs. Note that some cases may produce multiple follow-up inputs and outputs. - `relations`: Whether the output relation of the MR was satisfied for each follow-up output. - `verif_failure`: Whether any additional constraints were violated. `null` if all satisfied. ### RQ2 `rq2` compares the the metamorphic data with ground truths. `rq2/source-inputs-with-labels`: Contains the original test datasets used for the experiments (with ground truth labels). `rq2/source-output-correctness`: Contains lists of boolean values on whether the source outputs for a particular LLM-task pair aligns with the corresponding ground truths from the original test datasets. `rq2/ground_truth_comparison_summary.json`: A summary of the ground truth comparisons for each LLM-task-relation tuple. Each entry contains: - `llm_name`, `task_name`, `relation_name`: The target LLM, task, and MR, respectively.- `total`: The total number of valid metamorphic test cases run.- `mr_violated`: The total number of cases with metamorphic violations.- `mr_satisfied_gt_correct`, `mr_satisfied_gt_incorrect`, `mr_violated_gt_correct`, `mr_violated_gt_incorrect`: The number of cases where the MR was satisfied/violated and correct/incorrect according to the ground truth. ### RQ3 `rq3/manually_labelled_violations.json`: A collection of metamorphic violations, manually labelled as to the type of violation. The properties are: - `source_file`: The file name in `rq1/results` where the instance appears.- `llm_name`, `task_name`, `relation_name`: The used LLM, task, and MR, respectively.- `id`: The id of this instance found in the above `source_file`.- `source_input`, `source_output`, `followup_input`, `followup_output`: The source and follow-up inputs and outputs.- `output_relation`: Whether the output relation of the MR was satisfied. Always `false` in this case.- `followup_index`: Some tests produce multiple follow-up inputs/outputs. Here, each individual input-output pair is regarded separately. This value indicated the nth follow-up input/output used as found in the `source_file`.- `true_output`: The ground truth, as found in the original test datasets.- `failure_type`: The manually labelled type of violation. Values are `0` (true failure), `1` (intput transformation failure), `2` (output comparison failure), `3` (failure due to nonsensical MR), `4` (other failures), `5` (failure to identify 'unknown' in QA), and `6` (failure to recognise equivalence in RE). ### RQE `rqe` looks at re-running failing metamorphic cases 9 extra times to test for flakiness. `rqe/results`: Contains the raw data of 10 runs of test cases. Each instance has: - `source_file`: The file name in `rq1/results` where the instance appears.- `llm_name`, `task_name`, `relation_name`: The used LLM, task, and MR, respectively.- `id`: The id of this instance found in the above `source_file`.- `followup_index`: Some tests produce multiple follow-up inputs/outputs. Here, each individual input-output pair is regarded separately. This value indicated the nth follow-up input/output used as found in the `source_file`.- `source_input`, `followup_input`: The source and follow-up inputs.- `data`: The list of 10 instances run. - `followup_input`, `followup_output`: The source and follow-up outputs. - `output_relation`: Whether the outponal constraints were violated. `null` if all satisfied. `rqe/fail_rates.json`: Contains the calcaulated metamorphic violation rate for each run as described above, grouped by LLM-task-MR tuple. `rqe/fail_rates_summarised.json`: Contains the frequency of runs with violation rates of `0%` to `100%`, by `10%` increments. Grouped by LLM-task-MR tuple. `manual_vs_flakiness_comparison.json`: Takes the instances with manually labelled violation types (from `rq3/manually_labelled_violations.json`) and joins with the re-run violation rates (from `rqe/fail_rates.json`). ### Summary `summary` contains the summary data for the experiments. `summary/data_summary_by_cid.json`: Contains various summarised properties, grouped by LLM-task-MR tuple. Each instance contains: - `llm_name`, `task_name`, `relation_name`: The target LLM, task, and MR, respectively.- `total_count`: The total number of valid metamorphic test cases run.- `fail_count`: The total number of cases with metamorphic violations.- `fail_rate`: The proportion of cases that failed.- `fail_type`: The number of manually analysed failing cases for each respecive fail type. See `rq3` above for details.- `rerun_counts`: The number of failing cases for each flakiness rate. `null` for GPT-4, as those cases were not re-run. See `rqe` above for details.- `ground_truth_comparison`: The number of cases where the MR was satisfied/violated and correct/incorrect according to the ground truth. See `rq2` above for details. `summary/data_summary_by_category.json`: Contains various properties as described above (except `rerun_counts` and `ground_truth_comparison`), summarised individually by LLM, task, and MR.