Test Suites from Test-Generation Tools (Test-Comp 2021)

Test Suites This file describes the contents of an archive of the 3rd Competition on Software Testing (Test-Comp 2021).<br> https://test-comp.sosy-lab.org/2021/ The competition was run by Dirk Beyer, LMU Munich, Germany.<br> More information is available in the following article:<br> Dirk Beyer. <em>Status Report on Software Testing: Test-Comp 2021.</em> In Proceedings of the 24th International Conference on Fundamental Approaches to Software Engineering (FASE 2021, Luxembourg, March 27 - April 1), 2021. Springer. Copyright (C) Dirk Beyer<br> https://www.sosy-lab.org/people/beyer/ SPDX-License-Identifier: CC-BY-4.0<br> https://spdx.org/licenses/CC-BY-4.0.html Contents <code>LICENSE.txt</code>: specifies the license <code>README.txt</code>: this file <code>witnessFileByHash/</code>: This directory contains test suites (witnesses for coverage). Each witness in this directory is stored in a file whose name is the SHA2 256-bit hash of its contents followed by the filename extension .zip. The format of each test suite is described on the format web page: https://gitlab.com/sosy-lab/software/test-format A test suite contains also metadata in order to relate it to the test problem for which it was produced. <code>witnessInfoByHash/</code>: This directory contains for each test suite (witness) in directory witnessFileByHash/ a record in JSON format (also using the SHA2 256-bit hash of the witness as filename, with .json as filename extension) that contains the meta data. <code>witnessListByProgramHashJSON/</code>: For convenient access to all test suites for a certain program, this directory represents a function that maps each program (via its SHA2256-bit hash) to a set of test suites (JSON records for test suites as described above) that the test tools have produced for that program. For each program for which test suites exist, the directory contains a JSON file (using the SHA2 256-bit hash of the program as filename, with .json as filename extension) that contains all JSON records for test suites for that program. A similar data structure was used by SV-COMP and is described in the following article:<br> Dirk Beyer. <em>A Data Set of Program Invariants and Error Paths.</em> In Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR 2019, Montreal, Canada, May 26-27), pages 111-115, 2019. IEEE.<br> https://doi.org/10.1109/MSR.2019.00026 Other Archives Overview over archives from Test-Comp 2021 that are available at Zenodo: https://doi.org/10.5281/zenodo.4459466 Witness store (containing the generated test suites) https://doi.org/10.5281/zenodo.4459470 Results (XML result files, log files, file mappings, HTML tables) https://doi.org/10.5281/zenodo.4459132 Test tasks, version testcomp21 https://doi.org/10.5281/zenodo.4317433 BenchExec, version 3.6 All benchmarks were executed for Test-Comp 2021 https://test-comp.sosy-lab.org/2021/<br> by Dirk Beyer, LMU Munich, based on the following components: https://gitlab.com/sosy-lab/test-comp/archives-2021 testcomp21-0-gdacd4bf https://gitlab.com/sosy-lab/software/sv-benchmarks testcomp21-0-gefea738258 https://gitlab.com/sosy-lab/software/benchexec 3.6-0-gb278ebbb https://gitlab.com/sosy-lab/benchmarking/competition-scripts testcomp21-0-g8339740 https://gitlab.com/sosy-lab/test-comp/bench-defs testcomp21-0-g9d532c9 Contact Feel free to contact me in case of questions: https://www.sosy-lab.org/people/beyer/测试套件 本文档描述了第三届软件测试竞赛（Test-Comp 2021）归档文件的内容。竞赛官方网站：https://test-comp.sosy-lab.org/2021/ 本次竞赛由德国慕尼黑路德维希-马克西米利安大学（LMU Munich）的德克·贝耶尔（Dirk Beyer）主办。 更多详细信息可参阅以下学术论文： 德克·贝耶尔. 《软件测试现状报告：Test-Comp 2021》。发表于第24届国际软件工程基础方法国际会议（FASE 2021，卢森堡，3月27日-4月1日），2021年，Springer出版社。版权所有© 德克·贝耶尔 作者个人主页：https://www.sosy-lab.org/people/beyer/ SPDX许可证标识：CC-BY-4.0 许可证详情可访问：https://spdx.org/licenses/CC-BY-4.0.html 归档内容 - `LICENSE.txt`：用于说明本数据集的许可证条款 - `README.txt`：本文档 - `witnessFileByHash/`：该目录包含测试套件（覆盖率验证见证文件）。目录内每个见证文件均以其内容的SHA2 256位哈希值（SHA2 256-bit hash）作为文件名，并以`.zip`作为扩展名。测试套件的格式详见格式说明页面：https://gitlab.com/sosy-lab/software/test-format 每个测试套件均包含元数据，用于关联其对应的待测试问题。 - `witnessInfoByHash/`：针对`witnessFileByHash/`目录下的每个测试套件（见证文件），该目录存储一份JSON（JavaScript Object Notation）格式的元数据记录。文件名同样采用见证文件的SHA2 256位哈希值（SHA2 256-bit hash），扩展名为`.json`。 - `witnessListByProgramHashJSON/`：为方便检索特定程序对应的所有测试套件，该目录实现了一个映射关系：以程序的SHA2 256位哈希值（SHA2 256-bit hash）为键，映射至该程序的所有测试套件集合（即前述的JSON格式测试套件元数据记录）。对于每个存在测试套件的程序，该目录均包含一个以程序SHA2 256位哈希值（SHA2 256-bit hash）命名、扩展名为`.json`的文件，其中存储了该程序对应的所有测试套件元数据记录。SV-COMP曾采用类似的数据结构，相关说明可参阅以下论文： 德克·贝耶尔. 《程序不变量与错误路径数据集》。发表于2019年IEEE/ACM第16届软件仓库挖掘国际会议（MSR 2019，加拿大蒙特利尔，5月26-27日），第111-115页，2019年，IEEE出版社。 DOI链接：https://doi.org/10.1109/MSR.2019.00026 其他归档文件 Zenodo平台上可获取的Test-Comp 2021系列归档文件概览：https://doi.org/10.5281/zenodo.4459466 1. 见证文件存储库（包含生成的测试套件）：https://doi.org/10.5281/zenodo.4459470 2. 竞赛结果（包含XML结果文件、日志文件、文件映射表、HTML表格）：https://doi.org/10.5281/zenodo.4459132 3. 测试任务（版本为testcomp21）：https://doi.org/10.5281/zenodo.4317433 4. BenchExec（版本3.6）：本次Test-Comp 2021竞赛的所有基准测试均基于该工具执行，竞赛官方网站：https://test-comp.sosy-lab.org/2021/ 本数据集由德国慕尼黑路德维希-马克西米利安大学的德克·贝耶尔基于以下组件构建： - https://gitlab.com/sosy-lab/test-comp/archives-2021 版本：testcomp21-0-gdacd4bf - https://gitlab.com/sosy-lab/software/sv-benchmarks 版本：testcomp21-0-gefea738258 - https://gitlab.com/sosy-lab/software/benchexec 版本：3.6-0-gb278ebbb - https://gitlab.com/sosy-lab/benchmarking/competition-scripts 版本：testcomp21-0-g8339740 - https://gitlab.com/sosy-lab/test-comp/bench-defs 版本：testcomp21-0-g9d532c9 联系方式 如有任何疑问，可通过以下作者主页联系：https://www.sosy-lab.org/people/beyer/