stanford-crfm/DSIR-filtered-pile-100M-short

--- license: mit language: - en size_categories: - 10M<n<100M --- # Dataset Card for DSIR-filtered-pile-100M-short ## Dataset Description - **Repository:** https://github.com/p-lambda/dsir - **Paper:** - **Point of Contact: Sang Michael Xie <[e-mail]>** ### Dataset Summary This dataset is a subset of The Pile, selected via the DSIR data selection method. The target distribution for DSIR is the Wikipedia and BookCorpus2 subsets of The Pile. ### Languages English (EN) ## Dataset Structure A train set is provided (102M examples) with a small validation and test set (50k examples each). This dataset is more suitable for training shorter LMs (128 or 256 tokens). ### Data Instances ``` {"contents":"what did that mean for the future of the Firm?\n\nPublication Date:\n\nDiscipline:\n\nSource:\n\nProduct number:\n\nLength:\n\nAlso Available in:\n\ndescription\n\nOn Tuesday March 15, 2011, all 1,200 global Partners of McKinsey & Co. gathered at the Gaylord National Hotel & Convention Center near Washington, DC for their annual Partners' conference. The atmosphere was tense as Partners, in addition to their normal agenda, discussed the Galleon Group insider-trading trial and the recent allegations against the Firm's former Managing Director, Rajat Gupta. Three months earlier Senior Partner, Anil Kumar, plead guilty to providing confidential information about McKinsey clients he served to Galleon Group founder Raj Rajaratnam. The McKinsey Partners were shocked and dismayed by the actions of Kumar, as well as the recent allegations against Gupta and were closely monitoring the situation. Could a former Managing Director","metadata":{"pile_set_name":"Pile-CC"},"id":103} ``` ### Data Fields ``` "contents": the text "metadata": contains information about the source of text that the text comes from. "id": Ignore - a non-unique identifier ``` ## Dataset Creation We train the importance weight estimator for DSIR from The Pile validation set, where the target is Wikipedia + BookCorpus2 + Gutenberg + Books3 and the raw data come from the rest of the data sources in The Pile. We first select 92.4M examples from non-Wikipedia and book data, then randomly select 5M from Wikipedia and 1.66M each from BookCorpus2, Gutenberg, and Books3. ### Source Data The Pile #### Initial Data Collection and Normalization We select data from The Pile, which comes in 30 random chunks. We reserve chunk 0 for validation purposes and only consider the last 29 chunks. We first divided the documents in The Pile into chunks of 128 words, according to whitespace tokenization. These chunks define the examples that we do data selection on, totaling 1.7B examples. Before DSIR, we first apply a manual quality filter (see paper for details) and only consider the examples that pass the filter. ### Dataset Curators Sang Michael Xie, Shibani Santurkar ### Citation Information Paper: <https://arxiv.org/abs/2302.03169> ``` @article{xie2023data, author = {Sang Michael Xie and Shibani Santurkar and Tengyu Ma and Percy Liang}, journal = {arXiv preprint arXiv:2302.03169}, title = {Data Selection for Language Models via Importance Resampling}, year = {2023}, } ```