Abrak/wikipedia-paragraph-embeddings-en-gist-complete

该数据集包含英文维基百科每篇文章的段落嵌入，这些嵌入是基于wikimedia/wikipedia数据集生成的，并使用了avsolatorio/GIST-small-Embedding-v0模型进行量化处理。数据集的结构设计旨在最小化存储和计算需求，同时覆盖维基百科的广度。每个数据实例包含一个ID和一个嵌入向量，嵌入向量由384个int8值组成。数据来源于2023年11月1日的英文维基百科快照，嵌入计算使用了sentence_transformers和GIST-small-Embedding-v0模型，并在Nvidia A40上进行了约20小时的处理。数据集继承了CC-BY-SA-4.0和GFDL许可。

This dataset contains paragraph embeddings for every article in English Wikipedia, based on the wikimedia/wikipedia datasets 20231101.en version. The embeddings were generated using the avsolatorio/GIST-small-Embedding-v0 model and quantized to int8. The dataset structure is designed to minimize storage and computational requirements while covering the breadth of Wikipedia. Each data instance includes the ID of a paragraph and its corresponding list of 384 int8 values. The data is sourced from the wikimedia/wikipedia dataset, with article text split into paragraphs. The embedding calculation was performed on an Nvidia A40, taking approximately 20 hours. The dataset inherits the CC-BY-SA-4.0 and GFDL licenses from the Wikipedia article text.