CohereLabs/fineweb-edu-100B-index

---
license: cc-by-nc-sa-4.0
---

This dataset contains the embeddings for the 100B variant of [fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), embedded with the Cohere Embed V3 model.

You can search on this dataset with just 500MB of memory using [DiskVectorIndex](https://github.com/cohere-ai/DiskVectorIndex).

# Installation & Usage

Get your free **Cohere API key** from [cohere.com](https://cohere.com). You must set this API key as an environment variable:
```
export COHERE_API_KEY=your_api_key
```

Install the package:
```
pip install DiskVectorIndex
```

You can then search via:
```python
from DiskVectorIndex import DiskVectorIndex

index = DiskVectorIndex("CohereLabs/fineweb-edu-100B-index")

while True:
query = input("\n\nEnter a question: ")
docs = index.search(query, top_k=3)
for doc in docs:
print(doc)
print("=========")
```

# License

Please observe the License for the [Fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu). The license displayed here is just for the embeddings.

许可证：CC-BY-NC-SA-4.0

本数据集包含[fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) 100B变体的嵌入向量，嵌入过程使用Cohere Embed V3模型完成。

您可借助仅需500MB内存的磁盘向量索引（DiskVectorIndex）对本数据集进行检索，相关工具详情可参见[DiskVectorIndex](https://github.com/cohere-ai/DiskVectorIndex)。

# 安装与使用

请从[cohere.com](https://cohere.com)获取免费的**Cohere API密钥**，并将其设置为环境变量：

export COHERE_API_KEY=your_api_key

安装依赖包：

pip install DiskVectorIndex

随后即可通过以下方式进行检索：
python
from DiskVectorIndex import DiskVectorIndex

index = DiskVectorIndex("CohereLabs/fineweb-edu-100B-index")

while True:
query = input("

Enter a question: ")
docs = index.search(query, top_k=3)
for doc in docs:
print(doc)
print("=========")

# 许可证说明

请遵守[Fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)的原始许可证。此处展示的许可证仅适用于本数据集的嵌入向量。