AmanPriyanshu/Dynamic-Topic-RedPajama-Data-1T-100k-SubSample-max-1k-tokens

该数据集是从RedPajama-1T样本数据集中抽取的100,000个样本，每个样本限制为1,024个标记，专门用于动态主题建模应用。数据集通过GPT-4o-mini生成三个层次的主题抽象，包括广泛的主题领域、具体的重点领域和详细的主题元素。该数据集适用于训练主题生成模型、评估主题建模方法、研究文档分类层次结构等。

This dataset represents a curated subset of the RedPajama-1T Sample dataset, specifically processed for dynamic topic modeling applications. It contains 100,000 samples from the original dataset, with each document limited to the first 1,024 tokens for consistent processing. The dataset uses GPT-4o-mini to generate three levels of topic abstraction, including broad subject domains, specific focus areas, and detailed thematic elements. It is suitable for training topic generation models, evaluating topic modeling approaches, studying document categorization hierarchies, and more.