Pika4028/low-resource-rag-indexes

该数据集是低资源RAG：维基百科FAISS索引（BGE-M3），包含基于2023年维基百科文章段落构建的FAISS索引，使用BAAI/bge-m3模型进行嵌入。源语料库来自CohereLabs/wikipedia-2023-11-embed-multilingual-v3。数据集覆盖四种低资源语言：印地语（hi，541,822个段落，2.1 GB）、马拉地语（mr，203,479个段落，795 MB）、尼泊尔语（ne，83,598个段落，327 MB）和迈蒂利语（mai，13,806个段落，54 MB）。每个语言文件夹包含两个文件：index.faiss（FAISS IndexFlatIP索引，维度为1024，使用L2归一化的BGE-M3向量）和dataset/（Arrow数据集，包含每个段落的_id、title和text字段），这些文件按行对齐。该数据集旨在支持低资源语言的检索增强生成（RAG）应用，提供高效的向量搜索功能。This dataset is Low-Resource RAG: Wikipedia FAISS Indexes (BGE-M3), which includes FAISS indexes built from Wikipedia 2023 passages embedded using the BAAI/bge-m3 model. The source corpus is from CohereLabs/wikipedia-2023-11-embed-multilingual-v3. The dataset covers four low-resource languages: Hindi (hi, 541,822 passages, 2.1 GB), Marathi (mr, 203,479 passages, 795 MB), Nepali (ne, 83,598 passages, 327 MB), and Maithili (mai, 13,806 passages, 54 MB). Each language folder contains two files: index.faiss (a FAISS IndexFlatIP index with 1024-dimensional L2-normalized BGE-M3 vectors) and dataset/ (an Arrow dataset with _id, title, and text fields per passage), aligned row-by-row. The dataset is designed to support retrieval-augmented generation (RAG) applications for low-resource languages, providing efficient vector search capabilities.