科技需求文档数据集

数据来源于谷歌学术搜索以及cnki知网两个学术资源检索网站，数据集中的全部数据均由爬虫系统从两个学术资源检索网站中爬取获得。主要爬取每个学术资源的标题和摘要等信息存入数据库，通过使用知网的API信息获取某一类学术资源的资源列表，例如使用API链接https://search.cnki.com.cn/Search去检索虚拟现实技术，可以从中获取有关虚拟现实技术的列表URL信息，并以JSON对象形式返回给后台进行接受，作为信息存入到数据库当中。之后通过保存的URL信息去获取每个学术资源的详细信息，并且可以通过设置学术资源的主题、学科、资源类型等限制来使得爬虫去爬取我们更需要的学术资源信息，将所需信息存储到学术资源信息表Scdata当中。利用Python的SQLAlchemy包，创建数据库链接，读取SQL数据，导入excel表当中，将数据以excel表的形式进行保存，进而辅助研究需求图谱的构建和意图识别模型。该数据使用实该数据使用4029GP机架式服务器采集。 该数据可以应用于从需求文档中挖掘用户意图、约束、目标等要求的关联关系，支持需求知识图谱的构建。The dataset is sourced from two academic resource retrieval platforms: Google Scholar and CNKI (China National Knowledge Infrastructure). All data in the dataset was collected by a crawler system from these two platforms. The crawler primarily extracts metadata such as titles and abstracts of academic resources and stores them in a database. Specifically, the CNKI API is used to retrieve academic resource lists: for example, accessing the API endpoint https://search.cnki.com.cn/Search to query "virtual reality technology" will return a list of relevant URLs in JSON format, which are then received by the backend and stored in the database. Subsequently, the saved URLs are used to fetch detailed information of each academic resource. The crawler can be configured with filters including subject topics, academic disciplines, and resource types to target more relevant academic resources, and the required information is stored in the academic resource information table named Scdata. Python's SQLAlchemy library is utilized to establish database connections, read SQL data, export the data to Excel spreadsheets, and save the dataset in Excel format, which facilitates the construction of requirement knowledge graphs and the development of intent recognition models. The dataset was collected using a 4029GP rack-mounted server. This dataset can be applied to mine the relational connections among user intents, constraints, goals and other requirements from requirement documents, and supports the construction of requirement knowledge graphs.

搜集汇总

背景与挑战

背景概述

该数据集来源于谷歌学术和CNKI知网，通过爬虫系统获取学术资源的标题和摘要等信息，并以Excel格式保存。它旨在辅助需求图谱构建和意图识别模型，可用于挖掘需求文档中的用户意图、约束和目标等关联关系。