Topic Detection and Tracking (TDT)

Dataset for the project TDT (Project URL: https://github.com/sahandv/TDT) This dataset contains all required to train the models for TDT project. The dataset includes abstracts, keywords, mapped concepts, and citations for 194937 cleaned data points from the Scopus dataset, originally from over 300k data points. FastText Model: FastText trained model on Dimensions and Scopus data. This is used for keyword search and concept mapping. Computer Science Ontology (CSO): The original data can be downloaded from https://cso.kmi.open.ac.uk/downloads . The uploaded version is the parent map of the CSO, acquired using DFS. Every node has a level 2 parent (not level 1 root parent_. This is intended to give us an idea about the high-level topic abstractions for each low-level concept (node). So, each node will have a list of parents (topics). Scopus Dataset: (AI-related articles, from the start to 2020) - Preprocessed: - - `keyword pre-processed for fasttext - nov14`: preprocessed scopus publication author keywords, lemmatised with "|" delimiters -- `citations with abstracts`: citations that have documents and abstracts -- `citations with abstracts supernodes_str_name`: same as above, with additional supernodes based on textual clusters. -- `mapped concepts for keywords`: Concepts (level 2 from CS ontology), mapped using author keywords. Instead of keywords, this can be used. -- `abstract_title method_b_3`: This data is omitted to avoid copyright issues. However, can be provided to fellow researchers privately if requested, for non-commercial usage. The data is preprocessed and lemmatised abstracts with Scopus IDs. -- `data with abstract`: Containing these columns from the dataset: PY,id,eid,TI,author. Can be used to double-check the ID and PY (publication year). - `Doc2Vec Model`: Model trained using the Scopus and Dimensions data for AI articles, and doc2vec, with 100 dimensions. - `Node2Vec mode`: 100 dimensional node2vec model, trained using the citations. Use the Scopus id to get embeddings for each node. - Embeddings: -- `concepts_node2vec_50D`: 50-dimensional concept embedding for each article. (Scopus ID is not given as an index, but the order of data is the same as `abstract_title method_b_3` and can be joined.) -- `abstracts_doc2vec_100D`: 100-dimensional abstract embedding for each article. (Scopus ID is not given as an index, but the order of data is the same as `abstract_title method_b_3` and can be joined.) _______________________________________________________________________________________________ Please note that this dataset is only for academic and personal use, and commercial use of this dataset is prohibited. 3rd Party licence - Some materials are recompiled Scopus and Dimensions material and may be subject to their licensing.TDT项目数据集（项目地址：https://github.com/sahandv/TDT） 本数据集包含TDT项目模型训练所需的全部数据。 本数据集源自超30万条原始数据，经清洗后得到194937条有效数据点，包含摘要、关键词、映射概念与引用信息。 FastText 模型： 基于Dimensions与Scopus数据训练得到的FastText模型，用于关键词检索与概念映射。 计算机科学本体（Computer Science Ontology, CSO）： 原始数据可从https://cso.kmi.open.ac.uk/downloads 下载。本次上传的版本为经深度优先搜索（DFS）获取的CSO父级映射表，每个节点均包含二级父节点（而非一级根父节点），旨在帮助理解每个低级概念（节点）对应的高层主题抽象。因此，每个节点将对应一组父主题列表。 Scopus 数据集：（2020年及之前的人工智能相关文献） - 预处理文件： - `keyword pre-processed for fasttext - nov14`：已针对FastText完成预处理的Scopus出版物作者关键词，采用"|"作为分隔符并经过词形还原处理。 - `citations with abstracts`：包含文献与摘要的引用数据 - `citations with abstracts supernodes_str_name`：与上述内容一致，额外添加了基于文本聚类得到的超节点 - `mapped concepts for keywords`：基于作者关键词映射得到的概念（取自计算机科学本体的二级概念），可替代关键词使用 - `abstract_title method_b_3`：因版权问题未纳入公开数据集，但可应非商业用途的研究人员请求私下提供。该数据为经过预处理与词形还原的摘要数据，包含Scopus ID。 - `data with abstract`：包含数据集以下字段：PY、id、eid、TI、author，可用于核对文献ID与出版年份（PY）。 - `Doc2Vec 模型`：基于Scopus与Dimensions数据集的人工智能相关文献训练得到的Doc2Vec模型，维度为100维。 - `Node2Vec 模型`：基于引用数据训练得到的100维Node2Vec模型，可通过Scopus ID获取每个节点的嵌入向量。 - 嵌入向量文件： - `concepts_node2vec_50D`：单篇文献的50维概念嵌入向量（未将Scopus ID作为索引，但数据顺序与`abstract_title method_b_3`一致，可进行关联拼接）。 - `abstracts_doc2vec_100D`：单篇文献的100维摘要嵌入向量（未将Scopus ID作为索引，但数据顺序与`abstract_title method_b_3`一致，可进行关联拼接）。 _______________________________________________________________________________________________ 请注意，本数据集仅可用于学术研究与个人用途，严禁商业使用。 第三方许可声明：部分素材为重新编译的Scopus与Dimensions数据，可能受其原有许可协议约束。