smcleish/retrofitting-llama-fineweb-edu-tokenized

---
license: apache-2.0
language:
- en
---
This is the [350b FineWeb-Edu sample](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu/tree/main/sample/350BT), tokenized with the Llama-3 tokenizer for [Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence](https://arxiv.org/abs/2511.07384)

Please see our paper and [model collection](https://huggingface.co/collections/tomg-group-umd/retrofitting-recurrence), for more information.

# Streaming the Dataset
You can use datatrove to efficiently stream the dataset. Note ParquetReader, returns rows with "text_key" and "id_key", as the field names.

```
from datatrove.pipeline.readers import ParquetReader

data_reader = ParquetReader("hf://datasets/smcleish/retrofitting-llama-fineweb-edu-tokenized/dataset", limit=1, text_key="input_ids", id_key="attention_mask")
for document in data_reader():
# do something with document
print(document)
```

# Contact
Please, feel free to contact us with any questions, or open a discussion thread.

# Citation
```
@article{mcleish2025teaching,
title={Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence},
author={Sean McLeish and Ang Li and John Kirchenbauer and Dayal Singh Kalra and Brian R. Bartoldson and Bhavya Kailkhura and Avi Schwarzschild and Jonas Geiping and Tom Goldstein and Micah Goldblum},
journal={arXiv preprint arXiv:2511.07384},
year={2025}
}

许可证：Apache-2.0
语言：
- 英语

本数据集为[350B粒度FineWeb-Edu采样子集](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu/tree/main/sample/350BT)，已通过Llama-3分词器完成分词，用于论文《通过改进型循环机制引导预训练语言模型（Pretrained Language Model）进行深度思考》(https://arxiv.org/abs/2511.07384)的相关研究。

如需了解更多详情，请参阅我们的论文与[模型合集](https://huggingface.co/collections/tomg-group-umd/retrofitting-recurrence)。

# 数据集流式加载
你可借助datatrove高效流式加载本数据集。请注意，ParquetReader会返回字段名为`text_key`与`id_key`的数据行。

from datatrove.pipeline.readers import ParquetReader

data_reader = ParquetReader("hf://datasets/smcleish/retrofitting-llama-fineweb-edu-tokenized/dataset", limit=1, text_key="input_ids", id_key="attention_mask")
for document in data_reader():
# do something with document
print(document)

# 联系方式
如有任何疑问，欢迎随时联系我们，或发起讨论帖。

# 引用格式

@article{mcleish2025teaching,
title={Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence},
author={Sean McLeish and Ang Li and John Kirchenbauer and Dayal Singh Kalra and Brian R. Bartoldson and Bhavya Kailkhura and Avi Schwarzschild and Jonas Geiping and Tom Goldstein and Micah Goldblum},
journal={arXiv preprint arXiv:2511.07384},
year={2025}
}