neural-bridge/rag-full-20000

Name: neural-bridge/rag-full-20000
Creator: neural-bridge
Published: 2024-02-05 18:24:39
License: 暂无描述

Hugging Face2024-02-05 更新2024-03-04 收录

下载链接：

https://hf-mirror.com/datasets/neural-bridge/rag-full-20000

下载链接

链接失效反馈

官方服务：

资源简介：

--- dataset_info: features: - name: clear_prompt dtype: string splits: - name: train num_bytes: 43183498.53262665 num_examples: 17433 - name: test num_bytes: 10797732.467373349 num_examples: 4359 download_size: 32335855 dataset_size: 53981231 task_categories: - question-answering language: - en size_categories: - 10K<n<100K license: apache-2.0 tags: - retrieval-augmented-generation --- # **Retrieval-Augmented Generation (RAG) Full 20000** **Retrieval-Augmented Generation (RAG) Full 20000 is an English dataset designed for RAG-optimized models, built by [Neural Bridge AI](https://www.neuralbridge.ai/), and released under [Apache license 2.0](https://www.apache.org/licenses/LICENSE-2.0.html).** ## **Dataset Description** #### Dataset Summary Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by allowing them to consult an external authoritative knowledge base before generating responses. This approach significantly boosts the models' ability to produce relevant, accurate, and context-specific output by extending their capabilities to specialized domains or an organization's internal data, without the need for retraining. RAG offers a cost-effective method to leverage the vast data processing power of LLMs, equipped with billions of parameters, for tasks such as question-answering, language translation, and sentence completion, ensuring that the output is always up-to-date and applicable to various contexts. RAG's importance lies in its potential to address the inherent challenges of LLMs, such as unpredictability in responses, reliance on static and potentially outdated training data, and the risk of disseminating incorrect or non-authoritative information. These issues can negatively affect user trust in AI-powered applications, making RAG's ability to guide LLMs toward authoritative sources for information retrieval invaluable. RAG has multiple benefits, including cost-effective implementation and maintenance, access to current information, improved user trust through accurate information and source attribution, and greater control for developers over the information retrieval process. This approach allows for the dynamic updating of LLMs with the latest research, statistics, or news, directly addressing the challenges of maintaining relevancy and accuracy in rapidly changing knowledge landscapes. Additionally, it empowers organizations to deploy generative AI more confidently across a wider range of applications, enhancing both the user experience and the reliability of AI-driven interactions. Retrieval-Augmented Generation (RAG) Full 20000 dataset is a sigle-feature dataset, with each entry containing a "clear_prompt" field, designed to help build RAG-optimized models. This data consists of 20000 entries, and the data is from [Falcon RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb), [gsm8k](https://huggingface.co/datasets/gsm8k), and [RAG Hallucination Dataset 1000](https://huggingface.co/datasets/neural-bridge/rag-hallucination-dataset-1000). ```python from datasets import load_dataset rag_full = load_dataset("neural-bridge/rag-full-20000") ``` #### Languages The text in the dataset is in English. The associated BCP-47 code is `en`. ## **Dataset Structure** #### Data Instances A typical data point comprises the "clear_prompt" field, which is the concatenation of "context" (optional), "question", and "answer" fields. The context is obtained from [Falcon RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) and [RAG Hallucination Dataset 1000](https://huggingface.co/datasets/neural-bridge/rag-hallucination-dataset-1000). The question and answer for each data point are neither obtained by [gsm8k](https://huggingface.co/datasets/gsm8k) nor generated by GPT-4. An example from the dataset looks like the following: ``` { clear_prompt: ... } ``` #### Data Fields - `clear_prompt`: A string consisting of a range of tokens. It includes the "context (optional)", "question", and "answer" fields between "##CONTEXT##", "##QUESTION##", and "##ANSWER##" tags respectively. #### Data Splits The data is split into a training and test set. The split sizes are as follow: | | Train | Test | | ----- | ------ | ---- | | RAG Full 20000 | 17433 | 4359 | ## Source Data The data points in the dataset are from the [Falcon RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb), [gsm8k](https://huggingface.co/datasets/gsm8k), and [RAG Hallucination Dataset 1000](https://huggingface.co/datasets/neural-bridge/rag-hallucination-dataset-1000) datasets. ## **Neural Bridge AI RAG Datasets Index** | Model | Link | | ----- | ------ | | RAG Full 20000 | [link](https://huggingface.co/datasets/neural-bridge/rag-full-20000) | | RAG Dataset 12000 | [link](https://huggingface.co/datasets/neural-bridge/rag-dataset-12000) | | RAG Dataset 1200 | [link](https://huggingface.co/datasets/neural-bridge/rag-dataset-1200) | | RAG Hallucination Dataset 1000 | [link](https://huggingface.co/datasets/neural-bridge/rag-hallucination-dataset-1000) | ## **License** This public extract is made available under [Apache license 2.0](https://www.apache.org/licenses/LICENSE-2.0.html). Users should also abide to the [Falcon RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb), [gsm8k](https://huggingface.co/datasets/gsm8k), and [RAG Hallucination Dataset 1000](https://huggingface.co/datasets/neural-bridge/rag-hallucination-dataset-1000) ToUs.

数据集信息：特征： - 名称：clear_prompt，数据类型：字符串划分集： - 名称：train（训练集），字节数：43183498.53262665，样本数：17433 - 名称：test（测试集），字节数：10797732.467373349，样本数：4359 下载大小：32335855 数据集总大小：53981231 任务类别： - 问答（question-answering）语言： - 英语（en）样本规模类别： - 10K<n<100K 许可协议：apache-2.0 标签： - 检索增强生成（retrieval-augmented-generation） # **检索增强生成（Retrieval-Augmented Generation，RAG）全量20000数据集** **检索增强生成（Retrieval-Augmented Generation，RAG）全量20000是一款专为RAG优化模型设计的英语数据集，由[Neural Bridge AI](https://www.neuralbridge.ai/)构建，并依据[Apache协议2.0](https://www.apache.org/licenses/LICENSE-2.0.html)发布。** ## **数据集描述** #### 数据集摘要检索增强生成（RAG）通过允许大语言模型（Large Language Model，LLM）在生成回复前查阅外部权威知识库，以此增强其能力。该方法无需重新训练即可拓展模型到专业领域或组织内部数据场景，显著提升模型生成相关、准确且贴合上下文输出的能力，为依托数十亿参数的大语言模型的海量数据处理能力赋能问答、机器翻译、语句补全等任务，确保输出始终保持时效性且适配多样场景。 RAG的核心价值在于其有望解决大语言模型固有的诸多挑战，例如回复不可预测性、依赖静态且可能过时的训练数据，以及传播错误或非权威信息的风险。这些问题会损害用户对AI应用的信任，而RAG能够引导大语言模型从权威来源获取信息，其这一能力显得弥足珍贵。 RAG具备多重优势：实施与维护成本低廉、可获取最新信息、通过准确信息与来源归因提升用户信任度，同时让开发者对信息检索流程拥有更强掌控力。该方法支持为大语言模型动态更新最新研究、统计数据或新闻内容，直接应对快速变化的知识领域中维持相关性与准确性的挑战。此外，它还能帮助组织更放心地在更多应用场景中部署生成式AI（Generative AI），优化用户体验并提升AI驱动交互的可靠性。检索增强生成（RAG）全量20000数据集为单特征数据集，每条数据均包含`clear_prompt`字段，旨在助力构建RAG优化模型。该数据集共包含20000条数据，其来源为[Falcon RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)、[gsm8k](https://huggingface.co/datasets/gsm8k)以及[RAG Hallucination Dataset 1000](https://huggingface.co/datasets/neural-bridge/rag-hallucination-dataset-1000)。 python from datasets import load_dataset rag_full = load_dataset("neural-bridge/rag-full-20000") #### 语言说明数据集文本均为英语，对应的BCP-47代码为`en`。 ## **数据集结构** #### 数据样例一条典型数据样本包含`clear_prompt`字段，该字段由“上下文（可选）”“问题”与“答案”字段拼接而成。上下文数据来源于[Falcon RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)与[RAG Hallucination Dataset 1000](https://huggingface.co/datasets/neural-bridge/rag-hallucination-dataset-1000)。每条数据对应的问题与答案既非来自[gsm8k](https://huggingface.co/datasets/gsm8k)，也非由GPT-4生成。一条数据集样例格式如下： { clear_prompt: ... } #### 数据字段说明 - `clear_prompt`：由一系列Token组成的字符串，分别以`##CONTEXT##`、`##QUESTION##`与`##ANSWER##`标签包裹“可选上下文”“问题”与“答案”字段。 #### 数据划分数据集分为训练集与测试集，划分规模如下： | | 训练集 | 测试集 | | ----- | ------ | ---- | | RAG Full 20000 | 17433 | 4359 | ## 源数据本数据集的数据点来源于[Falcon RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)、[gsm8k](https://huggingface.co/datasets/gsm8k)以及[RAG Hallucination Dataset 1000](https://huggingface.co/datasets/neural-bridge/rag-hallucination-dataset-1000)数据集。 ## **Neural Bridge AI RAG数据集索引** | 模型名称 | 链接 | | ----- | ------ | | RAG Full 20000 | [链接](https://huggingface.co/datasets/neural-bridge/rag-full-20000) | | RAG Dataset 12000 | [链接](https://huggingface.co/datasets/neural-bridge/rag-dataset-12000) | | RAG Dataset 1200 | [链接](https://huggingface.co/datasets/neural-bridge/rag-dataset-1200) | | RAG Hallucination Dataset 1000 | [链接](https://huggingface.co/datasets/neural-bridge/rag-hallucination-dataset-1000) | ## **许可协议** 本公开数据集依据[Apache协议2.0](https://www.apache.org/licenses/LICENSE-2.0.html)发布。使用者同时需遵守[Falcon RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)、[gsm8k](https://huggingface.co/datasets/gsm8k)以及[RAG Hallucination Dataset 1000](https://huggingface.co/datasets/neural-bridge/rag-hallucination-dataset-1000)的使用条款。

提供机构：

neural-bridge

原始信息汇总

数据集概述

数据集名称

Retrieval-Augmented Generation (RAG) Full 20000

数据集描述

Retrieval-Augmented Generation (RAG) Full 20000 是一个为优化 RAG 模型设计的英语数据集，由 Neural Bridge AI 构建，并基于 Apache 2.0 许可证发布。该数据集通过允许大型语言模型在生成响应前咨询外部权威知识库，显著提升了模型生成相关、准确和上下文特定输出的能力。