PSB2: The Second Program Synthesis Benchmark Suite

<strong>PSB2: The Second Program Synthesis Benchmark Suite Datasets</strong> Version 1.0.1 (see version history at bottom) This repository contains datasets for the 25 problems described in the paper <em>PSB2: The Second Program Synthesis Benchmark Suite</em>. These problems come from a variety of sources, and require a range of programming constructs and datatypes to solve. These datasets are designed to be usable for any method of performing general program synthesis, including and not limited to inductive program synthesis and evolutionary methods such as genetic programming. For more information, see the associated website: https://cs.hamilton.edu/~thelmuth/PSB2/PSB2.html <strong>Use</strong> Each problem in the benchmark suite is located in a separate directory in the `datasets` directory. For each problem, we provide a set of `edge` cases and a set of `random` cases. The `edge` cases are hand-chosen cases representing the limits of the problem. The `random` cases are all generated based on problem-specific distributions. For each problem, we included exactly 1 million `random` cases. A typical use of these datasets for a set of runs of program synthesis would be: For each run, use every `edge` case in the training set For each run, use a different, randomly-sampled set of `random` cases in the training set. Use a larger set of `random` cases as an unseen test set. <strong>Sampling Libraries</strong> We provide the following libraries to make the downloading and sampling of these datasets easier. Using these libraries, you do not need to download the entire dataset from Zenodo; the individual problem datasets are downloaded and stored once when first sampling them. Python: https://github.com/thelmuth/psb2-python Clojure: https://github.com/thelmuth/psb2-clojure <strong>Dataset format</strong> Each edge and random dataset is provided in three formats: CSV, JSON, and EDN, with all three formats containing identical data. The CSV files are formatted as follows: The first row of the file is the column names. Each following row corresponds to one set of program inputs and expected outputs. Input columns are labeled `input1`, `input2`, etc., and output columns are labeled `output1`, `output2`, etc. In CSVs, string inputs and outputs are double quoted when necessary, but not if not necessary. Newlines within strings are escaped. Columns in CSV files are comma-separated. The JSON and EDN files are formatted using the JSON Lines standard (adapted for EDN).<br> Each case is put on its own line of the data file. The files should be read line-by-line and each parsed into an object/map using a JSON/EDN parser. <strong>Citation</strong> If you use these datasets in a publication, please cite the paper <em>PSB2: The Second Program Synthesis Benchmark Suite</em> and include a link to this repository. BibTeX entry for paper: <pre><code>@InProceedings{Helmuth:2021:GECCO, author = "Thomas Helmuth and Peter Kelly", title = "{PSB2}: The Second Program Synthesis Benchmark Suite", booktitle = "2021 Genetic and Evolutionary Computation Conference", series = {GECCO '21}, year = "2021", isbn13 = {978-1-4503-8350-9}, address = {Lille, France}, size = {10 pages}, doi = {10.1145/3449639.3459285}, publisher = {ACM}, publisher_address = {New York, NY, USA}, month = {10-14} # jul, doi-url = {https://doi.org/10.1145/3449639.3459285}, URL = {https://dl.acm.org/doi/10.1145/3449639.3459285}, }</code></pre> <strong>Version History</strong> 1.0.0 - 2021/4/10 - Initial publication of PSB2 datasets on Zenodo. 1.0.1 - 2021/7/9 - Changes to CSVs to quote all strings that could be read as integers. No changes in actual data, just formatting.

**PSB2：第二代程序合成基准测试集（PSB2: The Second Program Synthesis Benchmark Suite Datasets）** 版本1.0.1（版本历史见文末）
本仓库包含论文《PSB2：第二代程序合成基准测试集》中所述的25个问题对应的数据集。这些问题来源多样，求解需用到各类编程结构与数据类型。本数据集套件旨在适配任意通用程序合成方法，包括但不限于归纳式程序合成（inductive program synthesis）与遗传编程（genetic programming）等进化式方法。更多信息请访问官方网站：https://cs.hamilton.edu/~thelmuth/PSB2/PSB2.html

### 使用说明
基准测试集中的每个问题对应`datasets`目录下的一个独立子目录。针对每个问题，我们提供了`edge`（边界）用例集与`random`（随机）用例集。其中`edge`用例为人工甄选的、代表问题求解边界的测试用例；`random`用例则基于问题专属的分布规则生成。每个问题均包含恰好100万条`random`用例。
这类数据集在程序合成实验中的典型用法为：
1. 每次实验均使用训练集中的全部`edge`用例；
2. 每次实验从训练集中随机采样一组不同的`random`用例；
3. 使用规模更大的`random`用例集作为未见过的测试集。

### 采样工具库
我们提供了以下工具库以简化数据集的下载与采样流程。使用这些库时，无需从Zenodo（学术数据仓储平台）下载完整数据集，首次采样时仅会下载并存储单个问题的数据集。
- Python库：https://github.com/thelmuth/psb2-python
- Clojure库：https://github.com/thelmuth/psb2-clojure

### 数据集格式
所有`edge`与`random`数据集均提供CSV、JSON与EDN三种格式，三种格式包含完全一致的数据。
CSV文件格式说明如下：
文件首行为列名。后续每一行对应一组程序输入与预期输出。输入列以`input1`、`input2`等格式命名，输出列则以`output1`、`output2`等格式命名。在CSV文件中，字符串类型的输入与输出在必要时会使用双引号包裹，无需包裹时则不使用。字符串内的换行符会被转义。CSV文件的列以逗号分隔。
JSON与EDN文件遵循JSON行格式标准（适配EDN格式）。每条用例独占数据文件中的一行。读取文件时应逐行读取，并使用JSON/EDN解析器将每行解析为对象/映射表。

### 引用说明
若您在学术出版物中使用本数据集，请引用论文《PSB2：第二代程序合成基准测试集》，并附上本仓库的链接。论文的BibTeX引用条目如下：
bibtex
@InProceedings{Helmuth:2021:GECCO,
author = "Thomas Helmuth and Peter Kelly",
title = "{PSB2}: The Second Program Synthesis Benchmark Suite",
booktitle = "2021 Genetic and Evolutionary Computation Conference",
series = {GECCO '21},
year = "2021",
isbn13 = {978-1-4503-8350-9},
address = {Lille, France},
size = {10 pages},
doi = {10.1145/3449639.3459285},
publisher = {ACM},
publisher_address = {New York, NY, USA},
month = {10-14} # jul,
doi-url = {https://doi.org/10.1145/3449639.3459285},
URL = {https://dl.acm.org/doi/10.1145/3449639.3459285},
}

### 版本历史
1.0.0 - 2021/4/10：PSB2数据集首次在Zenodo平台发布。
1.0.1 - 2021/7/9：调整CSV文件格式，为所有可能被识别为整数的字符串添加引号。数据内容无任何变更，仅格式调整。