NotHotTryHard/wikipedia-en-harrier-270m-emb

--- dataset_info: features: - name: chunk_id dtype: int64 - name: article_title dtype: string - name: text dtype: string - name: embedding sequence: dtype: float32 length: 384 license: cc-by-sa-4.0 task_categories: - feature-extraction - text-retrieval tags: - wikipedia - embeddings - dense-retrieval - fact-checking - faiss - harrier language: - en size_categories: - 10M<n<100M pretty_name: Wikipedia EN Chunks + Harrier 270M Embeddings --- # Wikipedia EN Chunks + Harrier 270M Embeddings Pre-computed dense embeddings for **23.7M English Wikipedia chunks** using [microsoft/harrier-oss-v1-270m](https://huggingface.co/microsoft/harrier-oss-v1-270m) (384-dim). ## Related Datasets | Dataset | Description | |---|---| | [NotHotTryHard/wikipedia-en-harrier-270m-emb](https://huggingface.co/datasets/NotHotTryHard/wikipedia-en-harrier-270m-emb) | Same chunks, embedded with the larger **Harrier 270m** model | | [NotHotTryHard/wikipedia-en-harrier-0.6b-emb](https://huggingface.co/datasets/NotHotTryHard/wikipedia-en-harrier-0.6b-emb) | Same chunks, embedded with the larger **Harrier 0.6B** model | ## Dataset Details ### Source - **Wikipedia dump**: [wikimedia/wikipedia 20231101.en](https://huggingface.co/datasets/wikimedia/wikipedia) (6.4M articles) - **Chunking**: 200-word sliding window, 50-word overlap, min 50 characters - **Total chunks**: ~23,758,035 ### Embeddings - **Model**: `microsoft/harrier-oss-v1-270m` - **Dimension**: 384 - **Normalization**: L2-normalized - **Precision**: float32 ### Schema | Column | Type | Description | |---|---|---| | `chunk_id` | int64 | Unique chunk identifier (sequential) | | `article_title` | string | Wikipedia article title | | `text` | string | Chunk text (~200 words) | | `embedding` | list[float32] x 384 | L2-normalized dense vector | ### Storage - **Format**: Parquet shards with ZSTD compression - **Naming**: `data/train-XXXXX-of-NNNNN.parquet` ## Usage ```python from datasets import load_dataset ds = load_dataset("NotHotTryHard/wiki-en-harrier-270m", split="train") print(ds[0]) # {'chunk_id': 0, 'article_title': 'Anarchism', 'text': '...', 'embedding': [0.012, ...]} ``` ### Building a FAISS Index ```python import numpy as np import faiss embeddings = np.array(ds["embedding"], dtype=np.float32) index = faiss.IndexFlatIP(384) index.add(embeddings) ``` ## Pipeline ``` Wikipedia 20231101.en (6.4M articles) -> chunk.py (200w window, 50w overlap) -> 23.7M chunks in SQLite -> embed.py (harrier-oss-v1-270m, parallel GPU shards) -> export_parquet.py -> this dataset ``` ## License Wikipedia content is under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).dataset_info: features: - name: chunk_id dtype: int64 - name: article_title dtype: string - name: text dtype: string - name: embedding sequence: dtype: float32 length: 384 license: cc-by-sa-4.0 task_categories: - feature-extraction - text-retrieval tags: - wikipedia - embeddings - dense-retrieval - fact-checking - faiss - harrier language: - en size_categories: - 10M<n<100M pretty_name: 英文维基百科分块 + Harrier 270M 嵌入向量 # 英文维基百科分块 + Harrier 270M 嵌入向量 本数据集包含**2370万英文维基百科(Wikipedia)分块**的预计算稠密嵌入向量(embeddings)，采用[microsoft/harrier-oss-v1-270m](https://huggingface.co/microsoft/harrier-oss-v1-270m)模型生成，向量维度为384。 ## 相关数据集 | 数据集 | 描述 | |---|---| | [NotHotTryHard/wikipedia-en-harrier-270m-emb](https://huggingface.co/datasets/NotHotTryHard/wikipedia-en-harrier-270m-emb) | 采用更大规模的**Harrier 270m**模型对相同分块生成嵌入向量 | | [NotHotTryHard/wikipedia-en-harrier-0.6b-emb](https://huggingface.co/datasets/NotHotTryHard/wikipedia-en-harrier-0.6b-emb) | 采用更大规模的**Harrier 0.6B**模型对相同分块生成嵌入向量 | ## 数据集详情 ### 数据来源 - **维基百科转储文件(Wikipedia dump)**：[wikimedia/wikipedia 20231101.en](https://huggingface.co/datasets/wikimedia/wikipedia)（共640万篇文章） - **分块策略**：采用200词滑动窗口，窗口重叠50词，最小分块长度为50字符 - **总块数**：约23,758,035 ### 嵌入向量详情 - **生成模型**：`microsoft/harrier-oss-v1-270m` - **向量维度**：384 - **归一化方式**：L2归一化 - **精度格式**：float32 ### 数据结构 | 列名 | 数据类型 | 描述 | |---|---|---| | `chunk_id` | int64 | 唯一分块标识符（按序生成） | | `article_title` | 字符串 | 所属维基百科文章的标题 | | `text` | 字符串 | 分块文本（约200词） | | `embedding` | 384维float32列表 | L2归一化后的稠密向量 | ### 存储方式 - **存储格式**：采用ZSTD压缩的Parquet分块文件 - **文件命名规则**：`data/train-XXXXX-of-NNNNN.parquet` ## 使用示例 python from datasets import load_dataset ds = load_dataset("NotHotTryHard/wiki-en-harrier-270m", split="train") print(ds[0]) # {'chunk_id': 0, 'article_title': 'Anarchism', 'text': '...', 'embedding': [0.012, ...]} ### 构建FAISS索引 python import numpy as np import faiss embeddings = np.array(ds["embedding"], dtype=np.float32) index = faiss.IndexFlatIP(384) index.add(embeddings) ## 处理流程 维基百科 20231101.en（640万篇文章） -> chunk.py（200词窗口，50词重叠） -> SQLite中存储2370万条分块 -> embed.py（使用harrier-oss-v1-270m模型，并行GPU分块生成嵌入） -> export_parquet.py -> 生成本数据集 ## 许可证 维基百科内容遵循[CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)协议。