Package and Dependency Metadata for CZI Hackathon: Mapping the Impact of Research Software in Science

A collection of useful datasets extracted from https://packages.ecosyste.ms and https://repos.ecosyste.ms for use at the CZI Hackathon: Mapping the Impact of Research Software in Science. All data is provided as NDJSON (new line delimited JSON), each line represents a valid JSON object, and they are separated by newline characters. There are python and R libraries for reading these files, or you can maually read each line and parse each line as a single JSON object. Each ndjson file has been compressed with gzip (actual command: `tar -czvf`) to reduce download size, they expand to significantly bigger files after extraction. Package Data Package names from cran, bioconductor and pypi that have been parsed by the software-mentions project (data: https://datadryad.org/stash/dataset/doi:10.5061/dryad.6wwpzgn2c) are collected together with their latest release at time of publishing along with the names of their dependencies, those dependency names have then also been recursively fetched with latest release and dependencies until the full list of transitive dependencies is included. Note: This approach uses a simplified method of dependency resolution, always picking the latest version of each package rather than taking into account each dependencies specific version range requirements, this is primarily due to time constraints and allows all software ecosystems to be processed in the same way. A future improvement would be to use each package ecosystem's specific dependency resolution algorithm to compute the full transitive dependency tree for each mentioned software package. GitHub Data Two different approaches were taken for collecting data for referenced GitHub mentions: 1. `github.ndjson` is metadata for each repository from GitHub, including "manifest" files which are known files that contain dependency information for a project such as requirements.txt, DESCRIPTION and package.json, parsed using https://github.com/ecosyste-ms/bibliothecary, which may include transitive dependencies that have been discovered in a `lockfile` within the repository. 2. `github_packages.ndjson` is metadata for each package that was found on any package manager that references the GitHub url as it's repository url/source/homepage, these packages, like the cran and pypi data above, include the latest release and their direct dependencies. There may be more than one package for each GitHub URL as it is a one to many relationship. `github_packages_with_transitive.ndjson` follows the same format but also includes the extra resolved transitive dependencies of all packages using the same approach as with cran and pypi data above with the same caveats. There are also many more ecosystems referenced in these files than just cran, bioconductor and pypi, https://packages.ecosyste.ms provides a standardized metadata format for all of them to enable comparison and simplification of automation. Contact If you would like any help, support or more data from Ecosyste.ms please do get in touch via email: [e-mail] or open an issue on GitHub: https://github.com/ecosyste-ms/packages/issues本数据集为服务于陈-扎克伯格倡议（Chan Zuckerberg Initiative，CZI）黑客松「科研软件的科学影响力测绘」主题活动，从https://packages.ecosyste.ms与https://repos.ecosyste.ms提取得到的实用数据集集合。 所有数据均采用NDJSON（换行分隔JSON，New Line Delimited JSON）格式存储，每一行均为合法的JSON对象，行与行之间以换行符分隔。现有适用于Python和R语言的库可读取此类文件，您也可以手动逐行读取并将每行解析为单个JSON对象。 每个NDJSON文件均已使用gzip压缩（实际压缩命令为`tar -czvf`）以减小下载体积，解压后文件体积将显著增大。 ### 包数据 我们收集了由software-mentions项目（数据集来源：https://datadryad.org/stash/dataset/doi:10.5061/dryad.6wwpzgn2c）解析得到的CRAN（Comprehensive R Archive Network）、Bioconductor及PyPI（Python Package Index）的包名称，同时附上它们在数据集发布时的最新版本信息及其依赖项名称。随后我们会递归拉取这些依赖项的最新版本信息与依赖关系，直至涵盖完整的传递依赖列表。 注意：本方法采用了简化的依赖解析策略，始终选取每个包的最新版本，未考虑各依赖项的特定版本范围约束。这一设计主要受限于时间成本，但可让所有软件生态系统采用统一流程处理。未来的优化方向将是针对每个包生态系统使用其专属的依赖解析算法，为每个提及的软件包计算完整的传递依赖树。 ### GitHub数据 针对被引用的GitHub仓库数据，我们采用了两种采集方式： 1. `github.ndjson`：包含GitHub上每个仓库的元数据，其中涵盖了通过https://github.com/ecosyste-ms/bibliothecary解析得到的项目依赖信息文件（即所谓“清单文件”，如`requirements.txt`、`DESCRIPTION`和`package.json`），此类数据可能包含从仓库内`lockfile`中发现的传递依赖项。 2. `github_packages.ndjson`：收录了所有在包管理器中被发现、且以GitHub URL作为其仓库地址/来源/主页的包的元数据。与前述CRAN和PyPI数据类似，此类包包含其最新版本信息与直接依赖项。由于GitHub URL与包之间是一对多的映射关系，单个GitHub URL可能对应多个包。`github_packages_with_transitive.ndjson` 格式与前者一致，但额外包含了所有包的已解析传递依赖项，其解析逻辑与前述CRAN和PyPI数据的处理方式相同，且存在同样的局限性。 本数据集收录的生态系统远不止CRAN、Bioconductor和PyPI，https://packages.ecosyste.ms 为所有生态系统提供了标准化的元数据格式，以便于跨生态系统对比及自动化流程简化。 ### 联系方式 若您需要Ecosyste.ms的协助、支持或更多数据，请通过邮件联系：[e-mail]，或在GitHub提交Issue：https://github.com/ecosyste-ms/packages/issues