smcleish/retrofitting-llama-fineweb-edu-tokenized

---
license: apache-2.0
language:
- en
---
This is the [350b FineWeb-Edu sample](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu/tree/main/sample/350BT), tokenized with the Llama-3 tokenizer for [Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence](https://arxiv.org/abs/2511.07384)

Please see our paper and [model collection](https://huggingface.co/collections/tomg-group-umd/retrofitting-recurrence), for more information.

# Streaming the Dataset
You can use datatrove to efficiently stream the dataset. Note ParquetReader, returns rows with "text_key" and "id_key", as the field names.

```
from datatrove.pipeline.readers import ParquetReader

data_reader = ParquetReader("hf://datasets/smcleish/retrofitting-llama-fineweb-edu-tokenized/dataset", limit=1, text_key="input_ids", id_key="attention_mask")
for document in data_reader():
# do something with document
print(document)
```

# Contact
Please, feel free to contact us with any questions, or open a discussion thread.

# Citation
```
@article{mcleish2025teaching,
title={Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence},
author={Sean McLeish and Ang Li and John Kirchenbauer and Dayal Singh Kalra and Brian R. Bartoldson and Bhavya Kailkhura and Avi Schwarzschild and Jonas Geiping and Tom Goldstein and Micah Goldblum},
journal={arXiv preprint arXiv:2511.07384},
year={2025}
}