librarian-bots/pipeline-scripts

该数据集包含用于分类arXiv计算机科学论文并生成语义嵌入的脚本集合。主要功能包括使用ModernBERT模型识别引入新数据集的论文，以及使用BGE-base模型生成语义嵌入以支持向量搜索。输出数据集包括中间分类结果和最终带有嵌入的论文集合。

This dataset contains a collection of scripts for classifying arXiv computer science papers and generating semantic embeddings. Key functionalities include identifying papers that introduce new datasets using the ModernBERT model and generating semantic embeddings using the BGE-base model for vector search. Output datasets include intermediate classification results and the final collection of papers with embeddings.