PIPr: A Dataset of Public Infrastructure as Code Programs

Programming Languages Infrastructure as Code (PL-IaC) enables IaC programs written in general-purpose programming languages like Python and TypeScript. The currently available PL-IaC solutions are Pulumi and the Cloud Development Kits (CDKs) of Amazon Web Services (AWS) and Terraform. This dataset provides metadata and initial analyses of all public GitHub repositories in August 2022 with an IaC program, including their programming languages, applied testing techniques, and licenses. Further, we provide a shallow copy of the head state of those 7104 repositories whose licenses permit redistribution. The dataset is available under the Open Data Commons Attribution License (ODC-By) v1.0. Contents: metadata.zip: The dataset metadata and analysis results as CSV files. scripts-and-logs.zip: Scripts and logs of the dataset creation. LICENSE: The Open Data Commons Attribution License (ODC-By) v1.0 text. README.md: This document. redistributable-repositiories.zip: Shallow copies of the head state of all redistributable repositories with an IaC program. This artifact is part of the ProTI Infrastructure as Code testing project: https://proti-iac.github.io. Metadata The dataset's metadata comprises three tabular CSV files containing metadata about all analyzed repositories, IaC programs, and testing source code files. repositories.csv: ID (integer): GitHub repository ID url (string): GitHub repository URL downloaded (boolean): Whether cloning the repository succeeded name (string): Repository name description (string): Repository description licenses (string, list of strings): Repository licenses redistributable (boolean): Whether the repository's licenses permit redistribution created (string, date & time): Time of the repository's creation updated (string, date & time): Time of the last update to the repository pushed (string, date & time): Time of the last push to the repository fork (boolean): Whether the repository is a fork forks (integer): Number of forks archive (boolean): Whether the repository is archived programs (string, list of strings): Project file path of each IaC program in the repository programs.csv: ID (string): Project file path of the IaC program repository (integer): GitHub repository ID of the repository containing the IaC program directory (string): Path of the directory containing the IaC program's project file solution (string, enum): PL-IaC solution of the IaC program ("AWS CDK", "CDKTF", "Pulumi") language (string, enum): Programming language of the IaC program (enum values: "csharp", "go", "haskell", "java", "javascript", "python", "typescript", "yaml") name (string): IaC program name description (string): IaC program description runtime (string): Runtime string of the IaC program testing (string, list of enum): Testing techniques of the IaC program (enum values: "awscdk", "awscdk_assert", "awscdk_snapshot", "cdktf", "cdktf_snapshot", "cdktf_tf", "pulumi_crossguard", "pulumi_integration", "pulumi_unit", "pulumi_unit_mocking") tests (string, list of strings): File paths of IaC program's tests testing-files.csv: file (string): Testing file path language (string, enum): Programming language of the testing file (enum values: "csharp", "go", "java", "javascript", "python", "typescript") techniques (string, list of enum): Testing techniques used in the testing file (enum values: "awscdk", "awscdk_assert", "awscdk_snapshot", "cdktf", "cdktf_snapshot", "cdktf_tf", "pulumi_crossguard", "pulumi_integration", "pulumi_unit", "pulumi_unit_mocking") keywords (string, list of enum): Keywords found in the testing file (enum values: "/go/auto", "/testing/integration", "@AfterAll", "@BeforeAll", "@Test", "@aws-cdk", "@aws-cdk/assert", "@pulumi.runtime.test", "@pulumi/", "@pulumi/policy", "@pulumi/pulumi/automation", "Amazon.CDK", "Amazon.CDK.Assertions", "Assertions_", "HashiCorp.Cdktf", "IMocks", "Moq", "NUnit", "PolicyPack(", "ProgramTest", "Pulumi", "Pulumi.Automation", "PulumiTest", "ResourceValidationArgs", "ResourceValidationPolicy", "SnapshotTest()", "StackValidationPolicy", "Testing", "Testing_ToBeValidTerraform(", "ToBeValidTerraform(", "Verifier.Verify(", "WithMocks(", "[Fact]", "[TestClass]", "[TestFixture]", "[TestMethod]", "[Test]", "afterAll(", "assertions", "automation", "aws-cdk-lib", "aws-cdk-lib/assert", "aws_cdk", "aws_cdk.assertions", "awscdk", "beforeAll(", "cdktf", "com.pulumi", "def test_", "describe(", "github.com/aws/aws-cdk-go/awscdk", "github.com/hashicorp/terraform-cdk-go/cdktf", "github.com/pulumi/pulumi", "integration", "junit", "pulumi", "pulumi.runtime.setMocks(", "pulumi.runtime.set_mocks(", "pulumi_policy", "pytest", "setMocks(", "set_mocks(", "snapshot", "software.amazon.awscdk.assertions", "stretchr", "test(", "testing", "toBeValidTerraform(", "toMatchInlineSnapshot(", "toMatchSnapshot(", "to_be_valid_terraform(", "unittest", "withMocks(") program (string): Project file path of the testing file's IaC program Dataset Creation scripts-and-logs.zip contains all scripts and logs of the creation of this dataset. In it, executions/executions.log documents the commands that generated this dataset in detail. On a high level, the dataset was created as follows: A list of all repositories with a PL-IaC program configuration file was created using search-repositories.py (documented below). The execution took two weeks due to the non-deterministic nature of GitHub's REST API, causing excessive retries. A shallow copy of the head of all repositories was downloaded using download-repositories.py (documented below). Using analysis.ipynb, the repositories were analyzed for the programs' metadata, including the used programming languages and licenses. Based on the analysis, all repositories with at least one IaC program and a redistributable license were packaged into redistributable-repositiories.zip, excluding any node_modules and .git directories. Searching Repositories The repositories are searched through search-repositories.py and saved in a CSV file. The script takes these arguments in the following order: Github access token. Name of the CSV output file. Filename to search for. File extensions to search for, separated by commas. Min file size for the search (for all files: 0). Max file size for the search or * for unlimited (for all files: *). Pulumi projects have a Pulumi.yaml or Pulumi.yml (case-sensitive file name) file in their root folder, i.e., (3) is Pulumi and (4) is yml,yaml. https://www.pulumi.com/docs/intro/concepts/project/ AWS CDK projects have a cdk.json (case-sensitive file name) file in their root folder, i.e., (3) is cdk and (4) is json. https://docs.aws.amazon.com/cdk/v2/guide/cli.html CDK for Terraform (CDKTF) projects have a cdktf.json (case-sensitive file name) file in their root folder, i.e., (3) is cdktf and (4) is json. https://www.terraform.io/cdktf/create-and-deploy/project-setup Limitations The script uses the GitHub code search API and inherits its limitations: Only forks with more stars than the parent repository are included. Only the repositories' default branches are considered. Only files smaller than 384 KB are searchable. Only repositories with fewer than 500,000 files are considered. Only repositories that have had activity or have been returned in search results in the last year are considered. More details: https://docs.github.com/en/search-github/searching-on-github/searching-code The results of the GitHub code search API are not stable. However, the generally more robust GraphQL API does not support searching for files in repositories: https://stackoverflow.com/questions/45382069/search-for-code-in-github-using-graphql-v4-api Downloading Repositories download-repositories.py downloads all repositories in CSV files generated through search-respositories.py and generates an overview CSV file of the downloads. The script takes these arguments in the following order: Name of the repositories CSV files generated through search-repositories.py, separated by commas. Output directory to download the repositories to. Name of the CSV output file. The script only downloads a shallow recursive copy of the HEAD of the repo, i.e., only the main branch's most recent state, including submodules, without the rest of the git history. Each repository is downloaded to a subfolder named by the repository's ID.编程语言驱动的基础设施即代码（Programming Languages Infrastructure as Code，PL-IaC）支持使用通用编程语言（如Python、TypeScript）编写基础设施即代码（IaC）程序。当前已有的PL-IaC解决方案包括Pulumi，以及亚马逊云科技（Amazon Web Services, AWS）的云开发套件（Cloud Development Kits, CDKs）与Terraform。本数据集收录了2022年8月期间所有包含IaC程序的公开GitHub仓库的元数据与初步分析结果，涵盖其所用编程语言、采用的测试技术与开源许可证信息。此外，本数据集还为7104个许可证允许再分发的仓库提供了其提交头状态的浅克隆副本。本数据集采用开放数据共同体署名许可证（Open Data Commons Attribution License, ODC-By）v1.0进行授权。 Contents: metadata.zip：以CSV格式存储的数据集元数据与分析结果文件。 scripts-and-logs.zip：数据集构建过程所用脚本与运行日志。 LICENSE：开放数据共同体署名许可证（ODC-By）v1.0的官方文本。 README.md：本说明文档。 redistributable-repositiories.zip：所有符合再分发要求的IaC程序仓库的提交头状态浅克隆副本。 本数据集是ProTI基础设施即代码测试项目的组成部分，项目主页：https://proti-iac.github.io。 Metadata 本数据集的元数据包含三张结构化CSV表格，分别对应所有待分析仓库、IaC程序与测试源代码文件的元信息。 repositories.csv: ID（整数型）：GitHub仓库唯一标识符 url（字符串型）：GitHub仓库地址 downloaded（布尔型）：仓库克隆是否成功 name（字符串型）：仓库名称 description（字符串型）：仓库描述信息 licenses（字符串列表型）：仓库所采用的开源许可证 redistributable（布尔型）：仓库许可证是否允许再分发 created（日期时间字符串型）：仓库创建时间 updated（日期时间字符串型）：仓库最后更新时间 pushed（日期时间字符串型）：仓库最后推送时间 fork（布尔型）：该仓库是否为复刻仓库 forks（整数型）：仓库复刻数量 archive（布尔型）：该仓库是否已归档 programs（字符串列表型）：仓库内所有IaC程序的项目文件路径 programs.csv: ID（字符串型）：IaC程序的项目文件路径 repository（整数型）：所属GitHub仓库的唯一标识符 directory（字符串型）：IaC程序项目文件所在目录路径 solution（枚举字符串型）：IaC程序所用的PL-IaC解决方案，可选值为"AWS CDK"、"CDKTF"、"Pulumi" language（枚举字符串型）：IaC程序所用的编程语言，可选值包括："csharp"、"go"、"haskell"、"java"、"javascript"、"python"、"typescript"、"yaml" name（字符串型）：IaC程序名称 description（字符串型）：IaC程序描述信息 runtime（字符串型）：IaC程序的运行时配置字符串 testing（枚举字符串列表型）：IaC程序采用的测试技术，可选值包括："awscdk"、"awscdk_assert"、"awscdk_snapshot"、"cdktf"、"cdktf_snapshot"、"cdktf_tf"、"pulumi_crossguard"、"pulumi_integration"、"pulumi_unit"、"pulumi_unit_mocking" tests（字符串列表型）：IaC程序所有测试文件的路径 testing-files.csv: file（字符串型）：测试文件路径 language（枚举字符串型）：测试文件所用的编程语言，可选值包括："csharp"、"go"、"java"、"javascript"、"python"、"typescript" techniques（枚举字符串列表型）：测试文件中采用的测试技术，可选枚举值包括："awscdk"、"awscdk_assert"、"awscdk_snapshot"、"cdktf"、"cdktf_snapshot"、"cdktf_tf"、"pulumi_crossguard"、"pulumi_integration"、"pulumi_unit"、"pulumi_unit_mocking" keywords（枚举字符串列表型）：测试文件中检测到的关键词，可选枚举值包括："/go/auto"、"/testing/integration"、"@AfterAll"、"@BeforeAll"、"@Test"、"@aws-cdk"、"@aws-cdk/assert"、"@pulumi.runtime.test"、"@pulumi/"、"@pulumi/policy"、"@pulumi/pulumi/automation"、"Amazon.CDK"、"Amazon.CDK.Assertions"、"Assertions_"、"HashiCorp.Cdktf"、"IMocks"、"Moq"、"NUnit"、"PolicyPack("、"ProgramTest"、"Pulumi"、"Pulumi.Automation"、"PulumiTest"、"ResourceValidationArgs"、"ResourceValidationPolicy"、"SnapshotTest()"、"StackValidationPolicy"、"Testing"、"Testing_ToBeValidTerraform("、"ToBeValidTerraform("、"Verifier.Verify("、"WithMocks("、"[Fact]"、"[TestClass]"、"[TestFixture]"、"[TestMethod]"、"[Test]"、"afterAll("、"assertions"、"automation"、"aws-cdk-lib"、"aws-cdk-lib/assert"、"aws_cdk"、"aws_cdk.assertions"、"awscdk"、"beforeAll("、"cdktf"、"com.pulumi"、"def test_"、"describe("、"github.com/aws/aws-cdk-go/awscdk"、"github.com/hashicorp/terraform-cdk-go/cdktf"、"github.com/pulumi/pulumi"、"integration"、"junit"、"pulumi"、"pulumi.runtime.setMocks("、"pulumi.runtime.set_mocks("、"pulumi_policy"、"pytest"、"setMocks("、"set_mocks("、"snapshot"、"software.amazon.awscdk.assertions"、"stretchr"、"test("、"testing"、"toBeValidTerraform("、"toMatchInlineSnapshot("、"toMatchSnapshot("、"to_be_valid_terraform("、"unittest"、"withMocks(" program（字符串型）：该测试文件所属IaC程序的项目文件路径 Dataset Creation scripts-and-logs.zip包含本数据集构建过程的全部脚本与运行日志，其中executions/executions.log文件详细记录了生成本数据集所执行的所有命令。整体而言，本数据集的构建流程如下： 1. 通过search-repositories.py脚本（详见下文说明）生成所有包含PL-IaC程序配置文件的仓库列表。由于GitHub REST API存在非确定性特性，需要大量重试操作，因此该步骤耗时两周。 2. 通过download-repositories.py脚本（详见下文说明）下载所有仓库的提交头状态浅克隆副本。 3. 通过analysis.ipynb脚本对所有仓库进行分析，提取IaC程序的元数据，包括其所使用的编程语言与许可证信息。 4. 基于上述分析结果，将所有包含至少一个IaC程序且许可证允许再分发的仓库打包为redistributable-repositiories.zip，打包时将排除node_modules与.git目录。 Searching Repositories 仓库检索功能通过search-repositories.py脚本实现，检索结果将保存为CSV文件。该脚本需按以下顺序传入参数： 1. GitHub访问令牌 2. CSV输出文件名称 3. 待检索的目标文件名 4. 待检索的文件扩展名，多个扩展名以逗号分隔 5. 检索文件的最小尺寸（全文件检索时设为0） 6. 检索文件的最大尺寸，或传入*表示无限制（全文件检索时设为*） Pulumi项目在根目录下包含Pulumi.yaml或Pulumi.yml文件（文件名区分大小写），对应参数(3)为Pulumi，参数(4)为yml,yaml。详细说明可参考：https://www.pulumi.com/docs/intro/concepts/project/ AWS CDK项目在根目录下包含cdk.json文件（文件名区分大小写），对应参数(3)为cdk，参数(4)为json。详细说明可参考：https://docs.aws.amazon.com/cdk/v2/guide/cli.html CDK for Terraform (CDKTF)项目在根目录下包含cdktf.json文件（文件名区分大小写），对应参数(3)为cdktf，参数(4)为json。详细说明可参考：https://www.terraform.io/cdktf/create-and-deploy/project-setup Limitations 该脚本采用GitHub代码搜索API，因此继承了该API的固有局限性： - 仅收录复刻星标数大于原仓库的复刻仓库 - 仅分析仓库的默认分支 - 仅可检索尺寸小于384 KB的文件 - 仅收录文件总数少于500,000个的仓库 - 仅收录近一年内有活动记录或曾在搜索结果中出现过的仓库 更多细节可参考：https://docs.github.com/en/search-github/searching-on-github/searching-code GitHub代码搜索API的返回结果并不稳定。而通常稳定性更强的GraphQL API并不支持在仓库内检索文件，相关讨论可参考：https://stackoverflow.com/questions/45382069/search-for-code-in-github-using-graphql-v4-api Downloading Repositories download-repositories.py脚本可下载通过search-respositories.py生成的CSV文件中收录的所有仓库，并生成下载情况概览CSV文件。该脚本需按以下顺序传入参数： 1. 通过search-repositories.py生成的仓库CSV文件名称，多个文件以逗号分隔 2. 仓库下载目标目录 3. CSV输出文件名称 该脚本仅下载仓库HEAD提交的浅递归克隆副本，即仅下载主分支的最新状态（包含子模块），不包含完整Git提交历史。每个仓库将被下载至以其仓库ID命名的子目录中。