raghavnimbalkar/movie-screenplays-tokenized-dataset

该数据集包含约9400万token的专业格式剧本文本，已使用GPT-2分词器（openai-community/gpt2）进行预分词，并分为训练、验证和测试分区。原始源语料库从9个行业来源抓取的2500多个电影剧本中汇编而成，去重后约2400个独特标题，通过三个外部API丰富了元数据，按流派清洗和组织，然后分块为连续的512个token块。

This dataset contains approximately 94 million tokens of professionally formatted screenplay text, pre-tokenized using the GPT-2 tokenizer (openai-community/gpt2) and split into training, validation, and test partitions. The raw source corpus was assembled from 2,500+ movie scripts scraped across 9 industry sources, deduplicated to ~2,400 unique titles, enriched with metadata from three external APIs, cleaned and organized by genre, then tokenized into contiguous 512-token blocks.