guox18/formal-language-rnn-ppt

Formal Language RNN-PPT 是一个用于语言模型预预训练实验的数据集，包含由RNN生成的形式语言数据。该数据集已为EleutherAI/pythia-160m分词器进行标记化处理，包含100,000行数据，每行序列长度为2048，格式为Parquet，分割为训练集。每个行包括input_ids（标记ID）和attention_mask（注意力掩码）。数据集旨在用于预预训练，即在自然语言预训练之前，将随机初始化的语言模型首先暴露于结构化的合成形式语言数据中。完整版本包含333,217行数据，保存在full分支中。Formal Language RNN-PPT is a dataset for language model pre-pre-training experiments, which consists of formal language data generated by recurrent neural networks (RNNs). This dataset has been tokenized with the EleutherAI/pythia-160m tokenizer, and contains 100,000 rows of data with each sequence having a length of 2048. Stored in Parquet format, the dataset is split into a single training set. Each row includes `input_ids` (token IDs) and `attention_mask` (attention mask). The dataset is designed for pre-pre-training, i.e., to expose randomly initialized language models to structured synthetic formal language data prior to natural language pre-training. The full version of the dataset, which contains 333,217 rows of data, is stored in the `full` branch.