Results of the CT competition at the 11th International Workshop on Combinatorial Testing

These are the models and raw results achieved by participating tools in the CT competition at the 11th International Workshop on Combinatorial Testing (IWCT 2022).

The file ct-comp-models.tar.bz2 contains the input files in CTWedge and ACTS format.

The file ct-comp-final-results.tar.bz2 contains one folder per tool. In each of these folders, files conform to the following syntax:[toolname]_[model_filename]_[strength]_[iteration].[suffix]
where "toolname" is identical to the current folder,
"model_filename" is the exact name of the model file (including suffix),
"strength" is the combinatorial strength of the resulting CA (between 2 and 5),
"iteration" takes values between 1 and 3 and refers to the round of execution (at lower strengths, tools were tested using 3 iterations; at higher strengths, some tools required too much execution time and were only run once),
"suffix" is one of the following: "time" for the output of GNU time in verbose mode; "out" for the resulting CA (in headerless CSV form); "err" for the standard output (usually empty). Note that additional intermediate files may exist for some tools (e.g. when their original output is not strictly a headerless CSV).

All tools were executed in Docker containers based on the official OpenJDK image, which is itself based on Debian Buster (Docker identifier: openjdk:14-buster). The experiments were performed on a machine using a Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (16 physical cores, 32 logical cores) with 256 GB RAM.