castorini/NanoKnow-Fineweb-Edu-Index

---
license: apache-2.0
size_categories:
- 10M<n<100M
task_categories:
- text-retrieval
tags:
- lucene
- bm25
- fineweb
- nanochat
- information-retrieval
---

# NanoKnow FineWeb-Edu Lucene Index

[[Paper](https://huggingface.co/papers/2602.20122)] [[Code](https://github.com/castorini/NanoKnow)]

A pre-built [Lucene](https://lucene.apache.org/) BM25 index over [karpathy/fineweb-edu-100b-shuffle](https://huggingface.co/datasets/karpathy/fineweb-edu-100b-shuffle)—the exact pre-training corpus used by the [nanochat](https://github.com/karpathy/nanochat) family of language models. Built with [Anserini](https://github.com/castorini/anserini).

This index is part of the **NanoKnow** project: [github.com/castorini/NanoKnow](https://github.com/castorini/NanoKnow)

## Index Details

| Property | Value |
|----------|-------|
| **Corpus** | [karpathy/fineweb-edu-100b-shuffle](https://huggingface.co/datasets/karpathy/fineweb-edu-100b-shuffle) |
| **Documents** | 97,230,848 |
| **Index Size** | ~325 GB (extracted) |
| **Index Type** | Lucene (BM25) |
| **Built With** | Anserini / Pyserini |
| **Distribution** | 6 × `tar.part.*` files (~324 GB total), 680 Lucene segment files when extracted |

## Document ID Format

Each document has a unique ID: `shard_XXXXX_YYYYY`
- `XXXXX`: zero-padded shard number (0-1822)
- `YYYYY`: row offset within the parquet shard

For example, `shard_00151_20323` refers to row 20,323 in shard 151 of the FineWeb-Edu parquet files.

## Usage

### Download

The index is distributed as 6 split tar parts. Download all 6 parts and reassemble:

```bash
# Download all 6 parts (each ~64 GB; part.05 is ~4.4 GB)
for i in 00 01 02 03 04 05; do
wget https://huggingface.co/datasets/castorini/NanoKnow-Fineweb-Edu-Index/resolve/main/lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.$i
done

# (Optional) Verify checksums
md5sum -c <<'EOF'
309e75651d954a4d81edc6bc5b8f1d38 lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.00
313260d601b88ec443d2e7db94df08df lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.01
a2b446e7a40d89b1975c95f1abbd8683 lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.02
1e647f11aa01016a53f6c0847ce7ae86 lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.03
47a49ee4b2c7344b625e999c9658f817 lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.04
65ec80b055978356e5bd1772bdf18151 lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.05
EOF

# Reassemble + extract (streaming; never materializes the 325 GB tar on disk)
cat lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.* | tar -xf -

# This creates the directory:
# lucene-inverted.fineweb-edu-100b-karpathy.20260416/
```

Alternatively, you can use the Hugging Face CLI to fetch all 6 parts in one shot:

```bash
hf download castorini/NanoKnow-Fineweb-Edu-Index --repo-type dataset --local-dir ./fineweb-edu-index
cd ./fineweb-edu-index
cat lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.* | tar -xf -
```

### Search with Pyserini

```python
from pyserini.search.lucene import LuceneSearcher

searcher = LuceneSearcher("./lucene-inverted.fineweb-edu-100b-karpathy.20260416")
print(f"Index contains {searcher.num_docs:,} documents")

hits = searcher.search("What is the capital of France?", k=10)
for hit in hits:
print(f"{hit.docid}: {hit.score:.4f}")
```

### Retrieve Document Text

```python
import json

doc = searcher.doc("shard_00151_20323")
text = json.loads(doc.raw())["contents"]
print(text[:500])
```

## Reproducing BM25 Effectiveness

This index reproduces the published Anserini regression for NanoKnow v1 (NQ-Open
validation): **R@20 = 0.3283** with default BM25 (`k1=0.9, b=0.4`). See the
[Anserini documentation](https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/nanoknow-v1-nq.md)
for the full reproduction recipe.

## Related Resources

- **Benchmark Qrels**: [LingweiGu/NanoKnow_Benchmark](https://huggingface.co/datasets/LingweiGu/NanoKnow_Benchmark) — Pre-built relevance judgments that partition SQuAD and NQ questions into supported/unsupported splits based on this corpus.
- **Code**: [github.com/castorini/NanoKnow](https://github.com/castorini/NanoKnow) — Scripts to project new benchmarks onto this index, evaluate nanochat checkpoints, and analyze frequency effects.

## Citation

```bibtex
@article{gu2026nanoknow,
title={NanoKnow: How to Know What Your Language Model Knows},
author={Gu, Lingwei and Jedidi, Nour and Lin, Jimmy},
journal={arXiv preprint arXiv:2602.20122},
year={2026}
}
```

## License

Apache 2.0

---
许可证：Apache-2.0
规模类别：
- 1000万 < 样本数 < 1亿
任务类别：
- 文本检索
标签：
- Lucene
- BM25
- FineWeb
- nanochat
- 信息检索
---

# NanoKnow FineWeb-Edu Lucene 索引

[[论文]("https://huggingface.co/papers/2602.20122")] [[代码]("https://github.com/castorini/NanoKnow")]

本数据集为基于[karpathy/fineweb-edu-100b-shuffle]("https://huggingface.co/datasets/karpathy/fineweb-edu-100b-shuffle")构建的预构建Lucene BM25索引，该语料正是[nanochat]("https://github.com/karpathy/nanochat")系列大语言模型所使用的预训练语料。索引构建基于Anserini工具。

本索引属于**NanoKnow**项目的一部分：[github.com/castorini/NanoKnow]("https://github.com/castorini/NanoKnow")

## 索引详情

| 属性 | 取值 |
|----------|-------|
| **语料库** | [karpathy/fineweb-edu-100b-shuffle]("https://huggingface.co/datasets/karpathy/fineweb-edu-100b-shuffle") |
| **文档数** | 97,230,848 |
| **索引大小** | 约325 GB（解压后） |
| **索引类型** | Lucene（BM25） |
| **构建工具** | Anserini / Pyserini |
| **分发形式** | 6个`tar.part.*`分卷（总大小约324 GB），解压后包含680个Lucene索引段文件 |

## 文档ID格式

每个文档均拥有唯一ID，格式为`shard_XXXXX_YYYYY`：
- `XXXXX`：补零后的分片编号（取值范围0-1822）
- `YYYYY`：Parquet分片内的行偏移量

例如，`shard_00151_20323`指代FineWeb-Edu Parquet文件第151号分片中的第20323行数据。

## 使用方法

### 下载

本索引以6个Tar分卷形式分发，请下载全部6个分卷后合并：

bash
# 下载全部6个分卷（每个约64 GB；part.05约4.4 GB）
for i in 00 01 02 03 04 05; do
wget https://huggingface.co/datasets/castorini/NanoKnow-Fineweb-Edu-Index/resolve/main/lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.$i
done

# （可选）校验校验和
md5sum -c <<'EOF'
309e75651d954a4d81edc6bc5b8f1d38 lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.00
313260d601b88ec443d2e7db94df08df lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.01
a2b446e7a40d89b1975c95f1abbd8683 lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.02
1e647f11aa01016a53f6c0847ce7ae86 lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.03
47a49ee4b2c7344b625e999c9658f817 lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.04
65ec80b055978356e5bd1772bdf18151 lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.05
EOF

# 合并并解压（流式处理，无需在磁盘上生成完整的325 GB Tar文件）
cat lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.* | tar -xf -

# 解压后将生成以下目录：
# lucene-inverted.fineweb-edu-100b-karpathy.20260416/

或者，你也可以使用Hugging Face CLI一次性获取全部6个分卷：

bash
hf download castorini/NanoKnow-Fineweb-Edu-Index --repo-type dataset --local-dir ./fineweb-edu-index
cd ./fineweb-edu-index
cat lucene-inverted.fineweb-edu-100b-karpathy.20260416.tar.part.* | tar -xf -

### 使用Pyserini进行检索

python
from pyserini.search.lucene import LuceneSearcher

searcher = LuceneSearcher("./lucene-inverted.fineweb-edu-100b-karpathy.20260416")
print(f"索引包含 {searcher.num_docs:,} 个文档")

hits = searcher.search("What is the capital of France?", k=10)
for hit in hits:
print(f"{hit.docid}: {hit.score:.4f}")

### 检索文档文本

python
import json

doc = searcher.doc("shard_00151_20323")
text = json.loads(doc.raw())["contents"]
print(text[:500])

## 复现BM25检索效果

本索引可复现NanoKnow v1版本（NQ-Open验证集）中已发表的Anserini回归实验结果：采用默认BM25参数（`k1=0.9, b=0.4`）时，**R@20 = 0.3283**。完整复现流程请参考[Anserini官方文档]("https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/nanoknow-v1-nq.md")。

## 相关资源

- **基准查询相关性标注集**：[LingweiGu/NanoKnow_Benchmark]("https://huggingface.co/datasets/LingweiGu/NanoKnow_Benchmark") — 预构建的相关性判断集，可将SQuAD与NQ问题划分为基于该语料的支持/不支持子集。
- **代码库**：[github.com/castorini/NanoKnow]("https://github.com/castorini/NanoKnow") — 用于将新基准投影至本索引、评估nanochat模型checkpoint以及分析频率效应的脚本。

## 引用

bibtex
@article{gu2026nanoknow,
title={NanoKnow: How to Know What Your Language Model Knows},
author={Gu, Lingwei and Jedidi, Nour and Lin, Jimmy},
journal={arXiv preprint arXiv:2602.20122},
year={2026}
}

## 许可证

Apache 2.0