bergson-wikitext-2-4656-chunks

# bergson-wikitext-2-4656-chunks Pre-tokenized WikiText-2 in fixed 512-token GPT-2 chunks, used by the [bergson](https://github.com/EleutherAI/bergson) WikiText-2 / GPT-2 data-attribution replication of Bae et al. 2024 ([_Training Data Attribution via Approximate Unrolled Differentiation_](https://arxiv.org/abs/2405.12186)). Each row is one 512-token document (one block of the concatenated corpus). | Split | Rows | |-------|------| | `train` | 4656 | | `validation` | 481 | Columns: - `input_ids` — `list[int32]`, length 512 (GPT-2 BPE token ids) - `length` — `int64`, always 512 Load with `chunk_length: 0` in a bergson config (the data is already tokenized and chunked). ## How this was produced The standard HuggingFace `run_clm` preprocessing recipe — the same one kronfluence's `examples/wikitext` uses — applied to public [`Salesforce/wikitext`](https://huggingface.co/datasets/Salesforce/wikitext) `wikitext-2-raw-v1`: 1. Tokenize the raw `text` column with the GPT-2 tokenizer, no added special tokens. 2. `group_texts`: within each 1000-row `datasets.map` batch, concatenate the tokens, drop the remainder, and slice into fixed 512-token blocks. The 1000-row batching is load-bearing: it is what yields exactly 4656 / 481 chunks (whole-corpus concatenation instead gives 4671 / 482). No shuffling or seed is involved — the recipe is deterministic. ```python from datasets import DatasetDict, load_dataset from transformers import AutoTokenizer BLOCK = 512 tok = AutoTokenizer.from_pretrained("gpt2") raw = load_dataset("Salesforce/wikitext", "wikitext-2-raw-v1") def group_texts(examples): concat = sum(examples["input_ids"], []) total = (len(concat) // BLOCK) * BLOCK return {"input_ids": [concat[i : i + BLOCK] for i in range(0, total, BLOCK)]} out = {} for split in ("train", "validation"): toks = raw[split].map( lambda b: tok(b["text"]), batched=True, remove_columns=raw[split].column_names, ) blocks = toks.map( group_texts, batched=True, remove_columns=[c for c in toks.column_names if c != "input_ids"], ) out[split] = blocks.map(lambda x: {"length": len(x["input_ids"])}) DatasetDict(out).push_to_hub("EleutherAI/bergson-wikitext-2-4656-chunks") ``` This is also committed as `examples/replication/prep_dataset.py` in the bergson repo. The chunking reproduces the reference dataset byte-for-byte (identical `input_ids` on all 4656 train and 481 validation rows). ## License Inherits WikiText-2's CC BY-SA 3.0.