Complete Rxivist dataset of scraped bioRxiv data

rxivist.org allows readers to sort and filter the tens of thousands of preprints posted to bioRxiv. Rxivist uses a custom web crawler to index all papers on biorxiv.org; this is a snapshot of Rxivist the production database. The version number indicates the date on which the snapshot was taken. See the included "README.md" file for instructions on how to use the "rxivist.backup" file to import data into a PostgreSQL database server. Please note this is a different repository than the one used for the Rxivist manuscript—that is in a separate Zenodo repository. You're welcome (and encouraged!) to use this data in your research, but <strong>please cite our paper, now published in <em>eLife</em>.</strong> Going forward, this information will also be available pre-loaded into Docker images, available at blekhmanlab/rxivist_data. <strong>Version notes:</strong> <strong>2019-10-01</strong> The Crossref API is back online, and the "crossref_daily" table should now include up-to-date tweet information for July through September. About 40,000 authors were removed from the author table because the name had been removed from all preprints they had previously been associated with, likely because their name changed slightly on the bioRxiv website ("John Smith" to "J Smith" or "John M Smith"). The "author_emails" table was also modified to remove entries referring to the deleted authors. The web crawler is being updated to clean these orphaned entries more frequently. <strong>2019-08-30</strong> The Crossref Event Data API, which provides the data used to populate the table of tweet counts, has not been fully functional since early July. While we are optimistic that accurate tweet counts will be available at some point, the sparse values currently in the "crossref_daily" table for July and August should not be considered reliable. <strong>2019-07-01</strong> A new "institution" field has been added to the "article_authors" table that stores each author's institutional affiliation <em>as listed on that paper</em>. The "authors" table still has each author's most recently observed institution. We began collecting this data in the middle of May, but it has not been applied to older papers yet. <strong>2019-05-11</strong> The README was updated to correct a link to the Docker repository used for the pre-built images. <strong>2019-03-21</strong> The license for this dataset has been changed to CC-BY, which allows use for any purpose and requires only attribution. A new table, "publication_dates," has been added and will be continually updated. This table will include an entry for each preprint that has been published externally for which we can determine a date of publication, based on data from Crossref. (This table was previously included in the "paper" schema but was not updated after early December 2018.) Foreign key constraints have been added to almost every table in the database. This should not impact any read behavior, but anyone writing to these tables will encounter constraints on existing fields that refer to other tables. Most frequently, this means the "article" field in a table will need to refer to an ID that actually exists in the "articles" table. The "author_translations" table has been removed. This was used to redirect incoming requests for outdated author profile pages and was likely not of any functional use to others. The "README.md" file has been renamed "1README.md" because Zenodo only displays a preview for the file that appears first in the list alphabetically. The "article_ranks" and "article_ranks_working" tables have been removed as well; they were unused. <strong>2019-02-13.1</strong> After consultation with bioRxiv, the "fulltext" table will not be included in further snapshots until (and if) concerns about licensing and copyright can be resolved. The "docker-compose.yml" file was added, with corresponding instructions in the README to streamline deployment of a local copy of this database. <strong>2019-02-13</strong> The redundant "paper" schema has been removed. BioRxiv has begun making the full text of preprints available online. Beginning with this version, a new table ("fulltext") is available that contains the text of preprints that have been processed already. <strong>The format in which this information is stored may change in the future</strong>; any digression will be noted here. This is the first version that has a corresponding Docker image.rxivist.org 支持用户对上传至bioRxiv平台的数万篇预印本进行排序与筛选。Rxivist 采用定制化网络爬虫（web crawler）索引biorxiv.org上的全部论文，本数据集为Rxivist生产环境数据库的快照副本，版本号即该快照的生成日期。如需使用`rxivist.backup`文件向PostgreSQL数据库服务器导入数据，请参阅附带的`README.md`文件。请注意，本仓库与Rxivist相关论文所使用的仓库并非同一仓库——后者存储于独立的Zenodo仓库中。欢迎（并鼓励）您在研究中使用本数据集，但**请引用我们已发表于《eLife》的论文**。后续版本将预装至Docker镜像中，可通过blekhmanlab/rxivist_data获取。 <strong>版本说明：</strong> <strong>2019-10-01</strong> Crossref API已恢复正常运行，`crossref_daily`表现已可涵盖7月至9月的最新推文相关数据。约40000名作者被从作者表中移除，原因是其姓名已从所有此前关联的预印本中被移除——这大概率是由于他们在bioRxiv平台上的姓名发生了细微变更（例如“John Smith”变为“J Smith”或“John M Smith”）。`author_emails`表也同步进行了修改，移除了指向已删除作者的条目。我们正在更新网络爬虫，以更频繁地清理此类孤立条目。 <strong>2019-08-30</strong> 用于填充推文计数表的Crossref事件数据API自7月初起便未能完全正常运行。尽管我们乐观地认为未来可获取准确的推文计数，但当前`crossref_daily`表中7月和8月的稀疏数值不应被视为可靠数据。 <strong>2019-07-01</strong> 已在`article_authors`表中新增`institution`字段，用于存储**单篇论文中标注的**作者所属机构信息。`authors`表仍保留各作者最新记录的所属机构信息。我们于5月中旬开始采集该数据，但尚未将其应用于过往论文。 <strong>2019-05-11</strong> 更新了README文件，修正了预构建镜像所用Docker仓库的链接。 <strong>2019-03-21</strong> 本数据集的许可证已更改为CC-BY，该许可证允许任意用途的使用，仅需注明出处。新增了`publication_dates`表并将持续更新。该表将收录所有可通过Crossref数据确认外部发表日期的预印本条目。（该表此前曾包含在`paper`数据库模式中，但自2018年12月初起便未再更新。）已为数据库中几乎所有表添加外键约束。这不会影响任何读取操作，但任何向这些表写入数据的操作都将受到约束，要求现有字段引用其他表中实际存在的ID。最常见的情况是，某表中的`article`字段需引用`articles`表中实际存在的ID。已移除`author_translations`表，该表原本用于重定向对过时作者简介页面的请求，大概率对其他用户并无实际用途。已将`README.md`重命名为`1README.md`，这是因为Zenodo仅会按字母顺序显示列表中第一个文件的预览。同时移除了未使用的`article_ranks`与`article_ranks_working`表。 <strong>2019-02-13.1</strong> 经与bioRxiv协商，在解决许可与版权问题之前，后续快照将不再包含`fulltext`表。新增了`docker-compose.yml`文件，并在README中添加了相应说明，以简化本地数据库副本的部署流程。 <strong>2019-02-13</strong> 移除了冗余的`paper`数据库模式。bioRxiv已开始在线提供预印本的全文内容。自本版本起，新增了`fulltext`表，其中包含已处理的预印本文本。**本信息的存储格式未来可能发生变更**，任何变更都将在此处注明。这是首个拥有对应Docker镜像的版本。