PIPr: A Dataset of Public Infrastructure as Code Programs

Programming Languages Infrastructure as Code (PL-IaC) enables IaC programs written in general-purpose programming languages like Python and TypeScript. The currently available PL-IaC solutions are Pulumi and the Cloud Development Kits (CDKs) of Amazon Web Services (AWS) and Terraform. This dataset provides metadata and initial analyses of all public GitHub repositories in August 2022 with an IaC program, including their programming languages, applied testing techniques, and licenses. Further, we provide a shallow copy of the head state of those 7104 repositories whose licenses permit redistribution. The dataset is available under the Open Data Commons Attribution License (ODC-By) v1.0. Contents: metadata.zip: The dataset metadata and analysis results as CSV files. scripts-and-logs.zip: Scripts and logs of the dataset creation. LICENSE: The Open Data Commons Attribution License (ODC-By) v1.0 text. README.md: This document. redistributable-repositiories.zip: Shallow copies of the head state of all redistributable repositories with an IaC program. This artifact is part of the ProTI Infrastructure as Code testing project: https://proti-iac.github.io. Metadata The dataset's metadata comprises three tabular CSV files containing metadata about all analyzed repositories, IaC programs, and testing source code files. repositories.csv: ID (integer): GitHub repository ID url (string): GitHub repository URL downloaded (boolean): Whether cloning the repository succeeded name (string): Repository name description (string): Repository description licenses (string, list of strings): Repository licenses redistributable (boolean): Whether the repository's licenses permit redistribution created (string, date & time): Time of the repository's creation updated (string, date & time): Time of the last update to the repository pushed (string, date & time): Time of the last push to the repository fork (boolean): Whether the repository is a fork forks (integer): Number of forks archive (boolean): Whether the repository is archived programs (string, list of strings): Project file path of each IaC program in the repository programs.csv: ID (string): Project file path of the IaC program repository (integer): GitHub repository ID of the repository containing the IaC program directory (string): Path of the directory containing the IaC program's project file solution (string, enum): PL-IaC solution of the IaC program ("AWS CDK", "CDKTF", "Pulumi") language (string, enum): Programming language of the IaC program (enum values: "csharp", "go", "haskell", "java", "javascript", "python", "typescript", "yaml") name (string): IaC program name description (string): IaC program description runtime (string): Runtime string of the IaC program testing (string, list of enum): Testing techniques of the IaC program (enum values: "awscdk", "awscdk_assert", "awscdk_snapshot", "cdktf", "cdktf_snapshot", "cdktf_tf", "pulumi_crossguard", "pulumi_integration", "pulumi_unit", "pulumi_unit_mocking") tests (string, list of strings): File paths of IaC program's tests testing-files.csv: file (string): Testing file path language (string, enum): Programming language of the testing file (enum values: "csharp", "go", "java", "javascript", "python", "typescript") techniques (string, list of enum): Testing techniques used in the testing file (enum values: "awscdk", "awscdk_assert", "awscdk_snapshot", "cdktf", "cdktf_snapshot", "cdktf_tf", "pulumi_crossguard", "pulumi_integration", "pulumi_unit", "pulumi_unit_mocking") keywords (string, list of enum): Keywords found in the testing file (enum values: "/go/auto", "/testing/integration", "@AfterAll", "@BeforeAll", "@Test", "@aws-cdk", "@aws-cdk/assert", "@pulumi.runtime.test", "@pulumi/", "@pulumi/policy", "@pulumi/pulumi/automation", "Amazon.CDK", "Amazon.CDK.Assertions", "Assertions_", "HashiCorp.Cdktf", "IMocks", "Moq", "NUnit", "PolicyPack(", "ProgramTest", "Pulumi", "Pulumi.Automation", "PulumiTest", "ResourceValidationArgs", "ResourceValidationPolicy", "SnapshotTest()", "StackValidationPolicy", "Testing", "Testing_ToBeValidTerraform(", "ToBeValidTerraform(", "Verifier.Verify(", "WithMocks(", "[Fact]", "[TestClass]", "[TestFixture]", "[TestMethod]", "[Test]", "afterAll(", "assertions", "automation", "aws-cdk-lib", "aws-cdk-lib/assert", "aws_cdk", "aws_cdk.assertions", "awscdk", "beforeAll(", "cdktf", "com.pulumi", "def test_", "describe(", "github.com/aws/aws-cdk-go/awscdk", "github.com/hashicorp/terraform-cdk-go/cdktf", "github.com/pulumi/pulumi", "integration", "junit", "pulumi", "pulumi.runtime.setMocks(", "pulumi.runtime.set_mocks(", "pulumi_policy", "pytest", "setMocks(", "set_mocks(", "snapshot", "software.amazon.awscdk.assertions", "stretchr", "test(", "testing", "toBeValidTerraform(", "toMatchInlineSnapshot(", "toMatchSnapshot(", "to_be_valid_terraform(", "unittest", "withMocks(") program (string): Project file path of the testing file's IaC program Dataset Creation scripts-and-logs.zip contains all scripts and logs of the creation of this dataset. In it, executions/executions.log documents the commands that generated this dataset in detail. On a high level, the dataset was created as follows: A list of all repositories with a PL-IaC program configuration file was created using search-repositories.py (documented below). The execution took two weeks due to the non-deterministic nature of GitHub's REST API, causing excessive retries. A shallow copy of the head of all repositories was downloaded using download-repositories.py (documented below). Using analysis.ipynb, the repositories were analyzed for the programs' metadata, including the used programming languages and licenses. Based on the analysis, all repositories with at least one IaC program and a redistributable license were packaged into redistributable-repositiories.zip, excluding any node_modules and .git directories. Searching Repositories The repositories are searched through search-repositories.py and saved in a CSV file. The script takes these arguments in the following order: Github access token. Name of the CSV output file. Filename to search for. File extensions to search for, separated by commas. Min file size for the search (for all files: 0). Max file size for the search or * for unlimited (for all files: *). Pulumi projects have a Pulumi.yaml or Pulumi.yml (case-sensitive file name) file in their root folder, i.e., (3) is Pulumi and (4) is yml,yaml. https://www.pulumi.com/docs/intro/concepts/project/ AWS CDK projects have a cdk.json (case-sensitive file name) file in their root folder, i.e., (3) is cdk and (4) is json. https://docs.aws.amazon.com/cdk/v2/guide/cli.html CDK for Terraform (CDKTF) projects have a cdktf.json (case-sensitive file name) file in their root folder, i.e., (3) is cdktf and (4) is json. https://www.terraform.io/cdktf/create-and-deploy/project-setup Limitations The script uses the GitHub code search API and inherits its limitations: Only forks with more stars than the parent repository are included. Only the repositories' default branches are considered. Only files smaller than 384 KB are searchable. Only repositories with fewer than 500,000 files are considered. Only repositories that have had activity or have been returned in search results in the last year are considered. More details: https://docs.github.com/en/search-github/searching-on-github/searching-code The results of the GitHub code search API are not stable. However, the generally more robust GraphQL API does not support searching for files in repositories: https://stackoverflow.com/questions/45382069/search-for-code-in-github-using-graphql-v4-api Downloading Repositories download-repositories.py downloads all repositories in CSV files generated through search-respositories.py and generates an overview CSV file of the downloads. The script takes these arguments in the following order: Name of the repositories CSV files generated through search-repositories.py, separated by commas. Output directory to download the repositories to. Name of the CSV output file. The script only downloads a shallow recursive copy of the HEAD of the repo, i.e., only the main branch's most recent state, including submodules, without the rest of the git history. Each repository is downloaded to a subfolder named by the repository's ID.编程语言型基础设施即代码（Programming Languages Infrastructure as Code，PL-IaC）支持使用Python、TypeScript等通用编程语言编写的基础设施即代码（Infrastructure as Code，IaC）程序。当前已有的PL-IaC解决方案包括Pulumi，以及亚马逊云科技（Amazon Web Services，AWS）的云开发套件（Cloud Development Kits，CDKs）与Terraform。本数据集提供了2022年8月所有公开的、包含IaC程序的GitHub仓库的元数据与初步分析结果，涵盖其编程语言、应用的测试技术与许可协议。此外，我们提供了其中7104个许可允许再分发的仓库的HEAD状态浅拷贝。本数据集基于开放数据通用署名许可（Open Data Commons Attribution License，ODC-By）v1.0发布。 Contents: metadata.zip：以CSV文件形式存储的数据集元数据与分析结果。 scripts-and-logs.zip：数据集构建所用的脚本与日志。 LICENSE：开放数据通用署名许可（ODC-By）v1.0的文本文件。 README.md：本文档。 redistributable-repositiories.zip：所有符合再分发要求的IaC程序仓库的HEAD状态浅拷贝。 本成果隶属于ProTI基础设施即代码测试项目：https://proti-iac.github.io。 Metadata 本数据集的元数据包含三个表格形式的CSV文件，涵盖所有被分析仓库、IaC程序与测试源代码文件的元信息。 repositories.csv： ID（整数型）：GitHub仓库ID url（字符串型）：GitHub仓库URL downloaded（布尔型）：仓库克隆是否成功 name（字符串型）：仓库名称 description（字符串型）：仓库描述 licenses（字符串型，字符串列表）：仓库的许可协议 redistributable（布尔型）：仓库的许可协议是否允许再分发 created（字符串型，日期时间）：仓库创建时间 updated（字符串型，日期时间）：仓库最后更新时间 pushed（字符串型，日期时间）：仓库最后推送时间 fork（布尔型）：该仓库是否为派生仓库 forks（整数型）：派生仓库数量 archive（布尔型）：该仓库是否已归档 programs（字符串型，字符串列表）：仓库中每个IaC程序的项目文件路径 programs.csv： ID（字符串型）：IaC程序的项目文件路径 repository（整数型）：所属GitHub仓库ID directory（字符串型）：IaC程序项目文件所在目录路径 solution（字符串型，枚举值）：IaC程序所属的PL-IaC解决方案，可选值为"AWS CDK"、"CDKTF"、"Pulumi" language（字符串型，枚举值）：IaC程序所用编程语言，可选枚举值包括"csharp"、"go"、"haskell"、"java"、"javascript"、"python"、"typescript"、"yaml" name（字符串型）：IaC程序名称 description（字符串型）：IaC程序描述 runtime（字符串型）：IaC程序的运行时字符串 testing（字符串型，枚举值列表）：IaC程序所用的测试技术，可选枚举值包括"awscdk"、"awscdk_assert"、"awscdk_snapshot"、"cdktf"、"cdktf_snapshot"、"cdktf_tf"、"pulumi_crossguard"、"pulumi_integration"、"pulumi_unit"、"pulumi_unit_mocking" tests（字符串型，字符串列表）：IaC程序的测试文件路径 testing-files.csv： file（字符串型）：测试文件路径 language（字符串型，枚举值）：测试文件所用编程语言，可选枚举值包括"csharp"、"go"、"java"、"javascript"、"python"、"typescript" techniques（字符串型，枚举值列表）：测试文件中使用的测试技术，可选枚举值同前述testing字段的可选值 keywords（字符串型，枚举值列表）：测试文件中检出的关键词，可选枚举值包括"/go/auto"、"/testing/integration"、"@AfterAll"、"@BeforeAll"、"@Test"、"@aws-cdk"、"@aws-cdk/assert"、"@pulumi.runtime.test"、"@pulumi/"、"@pulumi/policy"、"@pulumi/pulumi/automation"、"Amazon.CDK"、"Amazon.CDK.Assertions"、"Assertions_"、"HashiCorp.Cdktf"、"IMocks"、"Moq"、"NUnit"、"PolicyPack("、"ProgramTest"、"Pulumi"、"Pulumi.Automation"、"PulumiTest"、"ResourceValidationArgs"、"ResourceValidationPolicy"、"SnapshotTest()"、"StackValidationPolicy"、"Testing"、"Testing_ToBeValidTerraform("、"ToBeValidTerraform("、"Verifier.Verify("、"WithMocks("、"[Fact]"、"[TestClass]"、"[TestFixture]"、"[TestMethod]"、"[Test]"、"afterAll("、"assertions"、"automation"、"aws-cdk-lib"、"aws-cdk-lib/assert"、"aws_cdk"、"aws_cdk.assertions"、"awscdk"、"beforeAll("、"cdktf"、"com.pulumi"、"def test_"、"describe("、"github.com/aws/aws-cdk-go/awscdk"、"github.com/hashicorp/terraform-cdk-go/cdktf"、"github.com/pulumi/pulumi"、"integration"、"junit"、"pulumi"、"pulumi.runtime.setMocks("、"pulumi.runtime.set_mocks("、"pulumi_policy"、"pytest"、"setMocks("、"set_mocks("、"snapshot"、"software.amazon.awscdk.assertions"、"stretchr"、"test("、"testing"、"toBeValidTerraform("、"toMatchInlineSnapshot("、"toMatchSnapshot("、"to_be_valid_terraform("、"unittest"、"withMocks(" program（字符串型）：测试文件所属IaC程序的项目文件路径 Dataset Creation scripts-and-logs.zip包含本数据集构建所用的全部脚本与日志，其中executions/executions.log详细记录了生成本数据集所执行的命令。从宏观层面来看，本数据集的构建流程如下： 1. 通过search-repositories.py脚本生成所有包含PL-IaC程序配置文件的仓库列表（脚本文档见下文）。由于GitHub REST API存在非确定性特征，需要大量重试，该执行过程耗时两周。 2. 通过download-repositories.py脚本下载所有仓库的HEAD状态浅拷贝（脚本文档见下文）。 3. 通过analysis.ipynb脚本对仓库进行分析，提取程序的元数据，包括所用编程语言与许可协议。 4. 根据分析结果，将所有包含至少一个IaC程序且许可协议允许再分发的仓库打包为redistributable-repositories.zip，打包过程中将排除所有node_modules与.git目录。 Searching Repositories 本数据集通过search-repositories.py脚本检索仓库并将结果保存至CSV文件。该脚本按顺序接收以下参数： 1. GitHub访问令牌 2. CSV输出文件名 3. 待检索的文件名 4. 以逗号分隔的待检索文件扩展名 5. 检索的最小文件大小（对所有文件设为0） 6. 检索的最大文件大小，或设为*表示无上限（对所有文件设为*） Pulumi项目在根目录下包含Pulumi.yaml或Pulumi.yml文件（文件名区分大小写），即对应上述参数(3)为Pulumi、参数(4)为yml,yaml。详见：https://www.pulumi.com/docs/intro/concepts/project/ AWS CDK项目在根目录下包含cdk.json文件（文件名区分大小写），即对应上述参数(3)为cdk、参数(4)为json。详见：https://docs.aws.amazon.com/cdk/v2/guide/cli.html CDK for Terraform（CDKTF）项目在根目录下包含cdktf.json文件（文件名区分大小写），即对应上述参数(3)为cdktf、参数(4)为json。详见：https://www.terraform.io/cdktf/create-and-deploy/project-setup Limitations 本脚本使用GitHub代码搜索API，因此继承了该API的局限性： 1. 仅包含星标数量多于父仓库的派生仓库 2. 仅考虑仓库的默认分支 3. 仅可检索大小小于384 KB的文件 4. 仅考虑包含少于500,000个文件的仓库 5. 仅考虑过去一年内有活动或曾在搜索结果中出现的仓库 更多详情：https://docs.github.com/en/search-github/searching-on-github/searching-code GitHub代码搜索API的结果并不稳定，而通常更为稳定的GraphQL API并不支持在仓库中检索文件：https://stackoverflow.com/questions/45382069/search-for-code-in-github-using-graphql-v4-api Downloading Repositories download-repositories.py脚本下载search-repositories.py生成的CSV文件中包含的所有仓库，并生成下载情况概览CSV文件。该脚本按顺序接收以下参数： 1. 由search-repositories.py生成的仓库CSV文件名，以逗号分隔 2. 仓库下载输出目录 3. CSV输出文件名 本脚本仅下载仓库HEAD的浅递归拷贝，即仅下载主分支的最新状态（包含子模块），不包含其余Git历史。每个仓库将被下载至以仓库ID命名的子文件夹中。