Binder Launch Records

The Binder group periodically releases a log of container launches from the public Binder service. Archives of launch records are available here, and a database version is available here. This repo includes a database dump with records from 2018-11-03 to 2021-01-21. These records do not include identifiable information like IP addresses, but do give the source repo being launched along with some other metadata. The database is stored in SQLite format (<code>binder-launches.sqlite</code>), with the following schema for the main table (<code>binder</code>). <pre><code>CREATE TABLE [binder] ( [timestamp] TEXT, [version] INTEGER, [provider] TEXT, [spec] TEXT, [ref] TEXT, [origin] TEXT, [repo] TEXT, [resolved_ref] TEXT, [org] TEXT );</code></pre> <code>timestamp</code> is the ISO timestamp of the launch <code>provider</code> gives the type of source repo being launched ("GitHub" is by far the most common). The rest of the explanations assume GitHub, other providers may differ. <code>spec</code> gives a unique-ish identifier for the particular branch/release being built. It consists of <code>&lt;github-id&gt;/&lt;repo&gt;/&lt;branch&gt;</code>. <code>ref</code>, <code>repo</code>, and <code>org</code> provide the same info, but split up differently (<code>repo</code> includes both the user ID/org and the actual repo name). These may be removed in a future release of this dataset, and shouldn't be used. <code>origin</code> indicates which backend was used. Each has its own storage, compute, etc. so this info might be important for evaluating caching and performance. <code>resolved_ref</code> specifies the git commit that was actually used, rather than the reference name. Note that this info was not recorded from the beginning, so only the more recent entries include it. The Binder launch dataset identifies the source repos that were used, but doesn't give any indication of their contents. We crawled GitHub to get the actual specification files in the repos which were fed into repo2docker when preparing the notebook environments, as well as filesystem metadata of the repos. Some repos were deleted/made private at some point, and were thus skipped. The results are in <code>binder-specs.sqlite</code>. The schema is as follows. <pre><code>CREATE TABLE specs ( ok BOOLEAN DEFAULT FALSE, remote TEXT NOT NULL, git_ref TEXT, git_commit TEXT, apt TEXT, conda TEXT, pip TEXT, pipfile TEXT, docker TEXT, setup TEXT, julia TEXT, r TEXT, nix TEXT, postbuild TEXT, start TEXT, runtime TEXT, ls text, resolved_commit TEXT, PRIMARY KEY(remote, git_ref, git_commit));</code></pre> The <code>ok</code> field indicates whether the repo was cloned successfully. It's probably fine to exclude any entries where <code>ok</code> is false from any processing. Here <code>remote</code> corresponds to <code>repo</code> in the launch database, <code>git_ref</code> to <code>ref</code>, and <code>git_commit</code> to <code>resolved_ref</code>. On newer records where the original dataset includes the resolved commit, it is included as part of the primary key. Note that since not all launch records include the resolved git commit, we couldn't simply use that as the primary key. In either case, the commit that was actually cloned is recorded in <code>resolved_commit</code>. For each repo, we collected spec files into the following fields (see the repo2docker docs for details on what these are). The records in the database are simply the verbatim file contents, with no parsing or further processing performed. <code>conda</code>: <code>environment.yml</code> <code>pip</code>: <code>requirements.txt</code> <code>apt</code>: <code>apt.txt</code> <code>pipfile</code>: <code>Pipfile.lock</code> or <code>Pipfile</code> <code>docker</code>: <code>Dockerfile</code> <code>setup</code>: <code>setup.py</code> <code>julia</code>: <code>Project.toml</code> or <code>REQUIRE</code> <code>r</code>: <code>install.R</code> <code>nix</code>: <code>default.nix</code> <code>postbuild</code>: <code>postBuild</code> <code>start</code>: <code>start</code> <code>runtime</code>: <code>runtime.txt</code> The <code>ls</code> field gives a metadata listing of the repo contents (excluding the <code>.git</code> directory). This field is JSON encoded with the following structure based on JSON types: Object: filesystem directory. Keys are file names within it. Values are the contents, which can be regular files, symlinks, or subdirectories. String: symlink. The string value gives the link target. Number: regular file. The number value gives the file size in bytes.