minishlab/tokenlearn-c4-en-bge-base-en-v1.5

--- language: en tags: - tokenlearn - embeddings - model2vec configs: - config_name: 10m data_files: - split: train path: 10m/train-* default: true --- # minishlab/tokenlearn-c4-en-bge-base-v1.5 Dataset Card This dataset was created with [Tokenlearn](https://github.com/MinishLab/tokenlearn) for training [Model2Vec](https://github.com/MinishLab/model2vec) models. It contains mean token embeddings produced by a sentence transformer, used as training targets for static embedding distillation. ## Dataset Details | Field | Value | |---|---| | **Source dataset** | [allenai/c4](https://huggingface.co/datasets/allenai/c4) | | **Source split** | `train` | | **Embedding model** | [baai/bge-base-en-v1.5](https://huggingface.co/baai/bge-base-en-v1.5) | | **Embedding dimension** | 768 | | **Rows** | 10000000 | ## Dataset Structure | Column | Type | Description | |---|---|---| | `text` | `string` | Truncated input text | | `embedding` | `list[float32]` | Mean token embedding from `baai/bge-base-en-v1.5`, excluding BOS/EOS tokens | ## Usage Load with the `datasets` library: ```python from datasets import load_dataset dataset = load_dataset("minishlab/tokenlearn-c4-en-bge-base-v1.5") ``` Train a Model2Vec model on this dataset using Tokenlearn: ```bash python -m tokenlearn.train \ --model-name "baai/bge-base-en-v1.5" \ --data-path "minishlab/tokenlearn-c4-en-bge-base-v1.5" \ --save-path "<path-to-save-model>" ``` ## Creation This dataset was created using the `tokenlearn-featurize` CLI: ```bash python -m tokenlearn.featurize \ --model-name "baai/bge-base-en-v1.5" \ --dataset-path "allenai/c4" \ --dataset-name "en" \ --dataset-split "train" \ --output-dir "<output-dir>" ``` ## Library Authors Tokenlearn was developed by the [Minish](https://github.com/MinishLab) team consisting of [Stephan Tulkens](https://github.com/stephantul) and [Thomas van Dongen](https://github.com/Pringled). ## Citation ``` @software{minishlab2024model2vec, author = {Stephan Tulkens and {van Dongen}, Thomas}, title = {Model2Vec: Fast State-of-the-Art Static Embeddings}, year = {2024}, publisher = {Zenodo}, doi = {10.5281/zenodo.17270888}, url = {https://github.com/MinishLab/model2vec}, license = {MIT} } ```--- language: 英语 tags: - tokenlearn - 词嵌入（embeddings） - model2vec configs: - config_name: 10m data_files: - split: train path: 10m/train-* default: true --- # minishlab/tokenlearn-c4-en-bge-base-v1.5 数据集卡片 本数据集通过Tokenlearn（Tokenlearn）创建，用于训练Model2Vec（Model2Vec）模型，其中包含由句子Transformer（sentence transformer）生成的平均Token（Token）嵌入，可用作静态嵌入蒸馏的训练目标。 ## 数据集详情 | 字段 | 取值 | |---|---| | **源数据集** | [allenai/c4](https://huggingface.co/datasets/allenai/c4) | | **源数据划分** | `train` | | **嵌入模型** | [baai/bge-base-en-v1.5](https://huggingface.co/baai/bge-base-en-v1.5) | | **嵌入维度** | 768 | | **数据行数** | 10000000 | ## 数据集结构 | 列名 | 数据类型 | 描述 | |---|---|---| | `text` | `字符串（string）` | 被截断的输入文本 | | `embedding` | `float32列表（list[float32]）` | 来自`baai/bge-base-en-v1.5`的平均Token嵌入，已剔除序列起始标记（BOS）与序列结束标记（EOS） | ## 使用方法 可通过`datasets`库加载本数据集： python from datasets import load_dataset dataset = load_dataset("minishlab/tokenlearn-c4-en-bge-base-v1.5") 随后可使用Tokenlearn在本数据集上训练Model2Vec模型，命令示例如下： bash python -m tokenlearn.train --model-name "baai/bge-base-en-v1.5" --data-path "minishlab/tokenlearn-c4-en-bge-base-v1.5" --save-path "<模型保存路径>" ## 数据集创建流程 本数据集通过`tokenlearn-featurize`命令行工具生成，具体命令如下： bash python -m tokenlearn.featurize --model-name "baai/bge-base-en-v1.5" --dataset-path "allenai/c4" --dataset-name "en" --dataset-split "train" --output-dir "<输出目录>" ## 开发团队 Tokenlearn由MinishLab团队开发，团队成员包括[Stephan Tulkens](https://github.com/stephantul)与[Thomas van Dongen](https://github.com/Pringled)。 ## 引用信息 bibtex @software{minishlab2024model2vec, author = {Stephan Tulkens 和 {van Dongen}, Thomas}, title = {Model2Vec：快速实现前沿静态词嵌入}, year = {2024}, publisher = {Zenodo}, doi = {10.5281/zenodo.17270888}, url = {https://github.com/MinishLab/model2vec}, license = {MIT} }