Reddit Corpus

Reddit是美国社交新闻汇总网站，用户可以在其中发布链接并参与有关这些帖子的讨论。这些线程讨论提供了一个很大的语料库，使用此目录中的工具将其转换为会话数据集。 每个reddit线程用于生成一组示例。每个响应注释都会生成一个示例，其中上下文是注释响应的注释的线性路径。如果注释或其直接父项具有超过128个字符或少于9个字符，则将过滤掉该示例。 来自注释父级的父级等的更多后退上下文被存储为额外的上下文功能。他们的文本被修剪成最多128个字符的长度，而不会分开单词。这有助于限制单个示例的大小。 训练/测试拆分是基于线程ID确定的。只要脚本的所有输入保持不变 (输入表，过滤阈值等)，生成的数据集应该是相同的。Reddit is a US-based social news aggregation website where users can post links and participate in discussions surrounding these posts. The thread discussions form a large corpus that can be converted into a conversational dataset using the tools provided in this directory. Each Reddit thread is used to generate a set of examples. Each response comment generates one example, where the context is the linear path of comments leading up to the responding comment. This example will be filtered out if either the comment or its direct parent has more than 128 characters or fewer than 9 characters. Additional backward context, such as the parent of the comment's parent and other further upstream context, is stored as an additional context feature. Their text is truncated to a maximum length of 128 characters without splitting words, which helps limit the size of individual examples. The train/test split is determined based on thread IDs. The generated dataset will remain consistent as long as all inputs to the script remain unchanged, e.g., input tables, filtering thresholds and other relevant parameters.

搜集汇总

背景与挑战

背景概述

Reddit Corpus是一个基于Reddit社交网站讨论线程构建的对话数据集，通过处理用户评论生成示例，并过滤掉不符合长度要求的上下文。该数据集适用于对话式响应选择和自然语言处理模型的预训练，其训练与测试分割基于线程ID以确保一致性。