LLMsForHepth/hep-th_hep-ph_gr-qc_primary

--- dataset_info: features: - name: id dtype: string - name: submitter dtype: string - name: authors dtype: string - name: title dtype: string - name: comments dtype: string - name: journal-ref dtype: string - name: doi dtype: string - name: report-no dtype: string - name: categories dtype: string - name: license dtype: string - name: orig_abstract dtype: string - name: versions list: - name: created dtype: string - name: version dtype: string - name: update_date dtype: string - name: authors_parsed sequence: sequence: string - name: abstract dtype: string splits: - name: train num_bytes: 443514798.708732 num_examples: 210905 - name: test num_bytes: 95039035.64563398 num_examples: 45194 - name: validation num_bytes: 95039035.64563398 num_examples: 45194 download_size: 355804687 dataset_size: 633592869.9999999 configs: - config_name: default data_files: - split: train path: data/train-* - split: test path: data/test-* - split: validation path: data/validation-* --- # Dataset Card for hep-th_hep-ph_gr-qc_primary Dataset ## Dataset Description - **Homepage:** [Kaggle arXiv Dataset Homepage](https://www.kaggle.com/Cornell-University/arxiv) - **Repository:** [hepthLlama](https://github.com/Paul-Richmond/hepthLlama) - **Paper:** [tbd](tbd) - **Point of Contact:** [Paul Richmond](mailto:[e-mail]) ### Dataset Summary This dataset contains metadata included in arXiv submissions. ## Dataset Structure An example from the dataset looks as follows: ``` {'id': '0908.2896', 'submitter': 'Paul Richmond', 'authors': 'Neil Lambert, Paul Richmond', 'title': 'M2-Branes and Background Fields', 'comments': '19 pages', 'journal-ref': 'JHEP 0910:084,2009', 'doi': '10.1088/1126-6708/2009/10/084', 'report-no': None, 'categories': 'hep-th', 'license': 'http://arxiv.org/licenses/nonexclusive-distrib/1.0/', 'abstract': ' We discuss the coupling of multiple M2-branes to the background 3-form and\n6-form gauge fields of eleven-dimensional supergravity, including the coupling\nof the Fermions. In particular we show in detail how a natural generalization\nof the Myers flux-terms, along with the resulting curvature of the background\nmetric, leads to mass terms in the effective field theory.\n', 'versions': [{'created': 'Thu, 20 Aug 2009 14:23:37 GMT', 'version': 'v1'}], 'update_date': '2009-11-09', 'authors_parsed': [['Lambert', 'Neil', ''], ['Richmond', 'Paul', '']]} ``` ### Languages The text in the `abstract` field of the dataset is in English, however there may be examples where the abstract also contains a translation into another language. ## Dataset Creation ### Curation Rationale The starting point was to load v193 of the Kaggle arXiv Dataset which includes arXiv submissions upto 23rd August 2024. The arXiv dataset contains the following data fields: - `id`: ArXiv ID (can be used to access the paper) - `submitter`: Who submitted the paper - `authors`: Authors of the paper - `title`: Title of the paper - `comments`: Additional info, such as number of pages and figures - `journal-ref`: Information about the journal the paper was published in - `doi`: [Digital Object Identifier](https://www.doi.org) - `report-no`: Report Number - `abstract`: The abstract of the paper - `categories`: Categories / tags in the ArXiv system To arrive at the hep-th_hep-ph_gr-qc_primary dataset, the full arXiv data was filtered so that only `categories` which included 'hep-th', 'hep-ph' or 'gr-qc' were retained. This resulted in papers that were either primarily classified as 'hep-th', 'hep-ph' or 'gr-qc' or appeared cross-listed. For this dataset, the decision was made to focus only on papers primarily classified as any of 'hep-th', 'hep-ph' or 'gr-qc'. This meant taking only those abstracts where the first characters in `categories` were any of 'hep-th', 'hep-ph' or 'gr-qc' (see [here](https://info.arxiv.org/help/arxiv_identifier_for_services.html#indications-of-classification) for more details). We also dropped entries whose `abstract` or `comments` contained the word 'Withdrawn' or 'withdrawn' and we removed the five records which appear in the repo `LLMsForHepth/arxiv_hepth_first_overfit`. In addition, we have cleaned the data appearing in `abstract` by first replacing all occurences of '\n' with a whitespace and then removing any leading and trailing whitespace. ### Data splits The dataset is split into a training, validation and test set with split percentages 70%, 15% and 15%. This was done by applying `train_test_split` twice (both with `seed=42`). The final split sizes are as follows: | Train | Test | Validation | |:---:|:---:|:---:| |210,905 | 45,194| 45,194 |数据集信息： 特征： - 名称：id，数据类型：字符串 - 名称：submitter（提交者），数据类型：字符串 - 名称：authors（作者），数据类型：字符串 - 名称：title（标题），数据类型：字符串 - 名称：comments（备注），数据类型：字符串 - 名称：journal-ref（期刊引用），数据类型：字符串 - 名称：doi（数字对象标识符，Digital Object Identifier），数据类型：字符串 - 名称：report-no（报告编号），数据类型：字符串 - 名称：categories（分类标签），数据类型：字符串 - 名称：license（许可协议），数据类型：字符串 - 名称：orig_abstract（原始摘要），数据类型：字符串 - 名称：versions（版本信息），列表类型： - 名称：created（创建时间），数据类型：字符串 - 名称：version（版本号），数据类型：字符串 - 名称：update_date（更新日期），数据类型：字符串 - 名称：authors_parsed（解析后的作者），序列类型（嵌套字符串序列） - 名称：abstract（摘要），数据类型：字符串 划分： - 名称：train（训练集），字节数：443514798.708732，样本数：210905 - 名称：test（测试集），字节数：95039035.64563398，样本数：45194 - 名称：validation（验证集），字节数：95039035.64563398，样本数：45194 下载大小：355804687，数据集总大小：633592869.9999999 配置： - 配置名称：default（默认配置），数据文件： - 划分：train，路径：data/train-* - 划分：test，路径：data/test-* - 划分：validation，路径：data/validation-* # hep-th_hep-ph_gr-qc_primary 数据集卡片 ## 数据集说明 - **主页**：[Kaggle arXiv数据集主页]("https://www.kaggle.com/Cornell-University/arxiv") - **代码仓库**：[hepthLlama]("https://github.com/Paul-Richmond/hepthLlama") - **相关论文**：[待更新](tbd) - **联系人**：[Paul Richmond]("mailto:[e-mail]") ### 数据集概述 本数据集收录arXiv（国际学术预印本平台）提交稿件的元数据。 ## 数据集结构 本数据集的一条示例样本如下： {'id': '0908.2896', 'submitter': 'Paul Richmond', 'authors': 'Neil Lambert, Paul Richmond', 'title': 'M2膜与背景场', 'comments': '19页', 'journal-ref': 'JHEP 0910:084,2009', 'doi': '10.1088/1126-6708/2009/10/084', 'report-no': 无, 'categories': 'hep-th（高能物理-理论，High Energy Physics - Theory）', 'license': 'http://arxiv.org/licenses/nonexclusive-distrib/1.0/', 'abstract': ' 本文讨论多组M2膜与十一维超引力的背景3-形式及6-形式规范场的耦合问题，其中包含费米子耦合部分。我们详细展示了Myers通量项的自然推广，结合背景度规的相应曲率，如何在有效场论中导出质量项。 ', 'versions': [{'created': '2009年8月20日 星期四 14:23:37 GMT', 'version': 'v1'}], 'update_date': '2009-11-09', 'authors_parsed': [['Lambert', 'Neil', ''], ['Richmond', 'Paul', '']]} ### 语言说明 数据集中`abstract`（摘要）字段的文本以英文为主，但部分样本的摘要可能同时包含其他语言的译文。 ## 数据集构建 ### 筛选依据 本数据集的原始数据源为Kaggle平台上的v193版arXiv数据集，该数据集收录了截至2024年8月23日的全部arXiv提交稿件。完整的arXiv数据集包含以下数据字段： - `id`：arXiv编号（可用于检索对应论文） - `submitter`：论文提交者 - `authors`：论文作者 - `title`：论文标题 - `comments`：补充信息，如页数、图表数量等 - `journal-ref`：论文发表期刊相关信息 - `doi`：数字对象标识符（Digital Object Identifier） - `report-no`：报告编号 - `abstract`：论文摘要 - `categories`：arXiv系统中的分类标签 为构建hep-th_hep-ph_gr-qc_primary数据集，我们对完整arXiv数据集进行了筛选，仅保留分类标签中包含`hep-th`（高能物理-理论，High Energy Physics - Theory）、`hep-ph`（高能物理-现象学，High Energy Physics - Phenomenology）或`gr-qc`（广义相对论与量子宇宙学，General Relativity and Quantum Cosmology）的稿件。最终得到的论文包括：主要分类为上述三类的稿件，以及被交叉归类的稿件。 本数据集仅保留主要分类为上述三类的论文，即仅选取`categories`字段的首段分类为`hep-th`、`hep-ph`或`gr-qc`的样本（详细分类规则可参考[此处]("https://info.arxiv.org/help/arxiv_identifier_for_services.html#indications-of-classification")）。 我们同时剔除了`abstract`或`comments`字段中包含“Withdrawn”或“withdrawn”字样的条目，并移除了`LLMsForHepth/arxiv_hepth_first_overfit`仓库中出现的5条记录。 此外，我们对`abstract`字段的数据进行了清洗：首先将所有换行符替换为空格，随后移除首尾多余的空白字符。 ### 数据划分 本数据集按照70%、15%、15%的比例划分为训练集、验证集与测试集。划分过程通过两次调用`train_test_split`工具实现（两次均设置随机种子`seed=42`）。最终划分的样本规模如下： | 训练集 | 测试集 | 验证集 | |:---:|:---:|:---:| |210,905 | 45,194| 45,194 |