Quantum Flakines Dynamic Testing

Detect flaky tests in quantum software using Dynamic approach Abstract This repository is a companion page for the paper “Detecting Flakiness in Quantum Software: A Dynamic Testing Approach.” The dataset contains two types of files: 1. Per-release execution-summary CSV files: 23 CSV files, one for each Qiskit Terra release studied. Each file summarizes the outcomes of 10,000 executions of the test suite for that release. 2. Root-cause and fix-labels file: one spreadsheet containing manually assigned root-cause and fix-pattern labels for the flaky tests identified in the study. Per-release CSV files Each of the 23 CSV files shares the following common schema. Attribute Description test_case_name The identifier of the individual. passed_attempts Number of runs in which the test passed. failed_attempts Number of runs in which the test failed. skipped_runs Number of skipped runs (e.g., via skip markers orenvironment constraints);skips were outside ourcontrol. total_runs The total number of executions attempted forthe test case; computed as passed_attempts +failed_attempts + skipped_runs; equals 10,000in all retained rows. success_rate The fraction of successful executions; computedas passed_attempts/total_runs. qiskit_version The terra release tag used for the run. python_version Python version used (always 3.8). rust_version Rust toolchain version used to satisfy terrabuild requirements for that release. Root-cause and fix-labels file In addition to the per-release execution summaries, the dataset includes a spreadsheet named qiskit_flaky_tests_root_cause_fix_labels.xlsx. This file contains the manually assigned root-cause and fix-pattern labels for the flaky tests identified in the study. Each row corresponds to one flaky test case and records the last release in which the test was observed to be flaky, whether and when it was fixed or removed, and the manually assigned root-cause and fix-pattern categories. The schema is as follows Attribute Description test_case_name Fully qualified identifier of the flaky test case. Each entry is hyperlinked to the corresponding GitHub source code for the test case, pointing to the latest Qiskit Terra version in which the test case was observed to be flaky. `last_flaky_version Last Qiskit Terra release in the study window in which the test was observed to be flaky Fixed in Qiskit release or release candidate in which the flaky behavior was fixed, when a fix could be identified. This field may be empty if no specific fix release was identified. Removed in Qiskit release or release candidate in which the test or related functionality was removed, when applicable. This field may be empty if the test was not removed. Root cause Manually assigned high-level root-cause category for the flaky behavior. Root cause sub-category More specific explanation of the root cause, refining the high-level category. Category of fix Manually assigned high-level category describing how the flaky behavior was addressed. Category of fix sub-category More specific explanation of the fix pattern, refining the high-level fix category.