xtenzr/dv-relevancy-10k

# DeVal Dataset ## Dataset Description ### Overview A specialized dataset designed for training and evaluating RAG (Retrieval-Augmented Generation) systems, focusing on key aspects of response quality assessment including relevancy, hallucination detection, completeness, and attribution accuracy. ## File Statistics | File | Examples | Size (MB) | Tasks | Avg Context Length | Avg Query Length | Avg Response Length | |------|----------|-----------|-------|-------------------|-----------------|-------------------| | batch_dv_relevancy_5006_train_binary.jsonl | 5,006 | 6.46 | relevancy:5006 | 127.3 | 13.7 | 64.4 | | batch_dv_relevancy_5006_validate_binary.jsonl | 5,000 | 6.47 | relevancy:5000 | 127.7 | 13.7 | 64.7 | |------|----------|-----------|-------|-------------------|-----------------|-------------------| | **TOTAL** | **10,006** | **12.93** | | | | | ### Supported Tasks The dataset supports the following evaluation tasks: - relevancy: 10,006 examples (100.0%) ### Dataset Statistics - Total examples: 10,006 - Format: JSONL - Size on disk: 12.9MB ### Data Fields - `task`: Type of evaluation task - `rag_context`: Source context for RAG system - `query`: User query/question - `llm_response`: Response to evaluate - `reference`: Reference score (0.0-1.0) ### Data Splits - Training: 100% (10,006 examples) - No predefined validation/test splits ## Dataset Structure ### Length Statistics (in tokens) #### Context - Minimum: 80 - Maximum: 193 - Mean: 127.5 - Median: 127.0 #### Queries - Minimum: 7 - Maximum: 31 - Mean: 13.69 - Median: 13.0 #### Responses - Minimum: 10 - Maximum: 171 - Mean: 64.57 - Median: 59.0 ### Reference Score Distribution | Score | Count | Percentage | |--------|--------|------------| | 0.0 | 5,092 | 50.9% | | 1.0 | 4,914 | 49.1% | ## Sample Examples ### Relevancy Examples #### Example 1 - Context: "TechServe Solutions is a growing IT support company that specializes in providing technical assistance for small to medium-sized businesses. The company has a robust knowledge base that includes troub..." - Query: "What are the best practices for remote troubleshooting in technical support?" - Response: "To improve your marketing strategy, consider investing in social media advertising, creating engaging content, and analyzing customer feedback. It's also essential to identify your target audience and tailor your campaigns accordingly." - Reference Score: 0.0 #### Example 2 - Context: "Our company specializes in providing cloud-based software solutions for small to medium-sized businesses. Recently, we introduced a new tiered pricing plan designed to accommodate varying needs, which..." - Query: "Can you explain the differences between the basic, professional, and enterprise pricing plans?" - Response: "The basic plan provides essential features suitable for startups and small businesses, including limited customer support and access to core functionalities. The professional plan expands on this with additional features, enhanced customer support, and analytics tools, ideal for growing businesses that require more robust solutions. The enterprise plan offers all the features of the previous plans, plus advanced security options, dedicated account management, and customized solutions tailored to large organizations that need scalable resources." - Reference Score: 1.0 #### Example 3 - Context: "XYZ Electronics has recently launched a new line of smart home devices, including smart lights, thermostats, and security cameras. As part of their customer support strategy, they have implemented a t..." - Query: "What can customers do if their smart thermostat keeps disconnecting from Wi-Fi?" - Response: "If your smart thermostat keeps disconnecting from Wi-Fi, here are a few troubleshooting steps you can try: First, ensure that your Wi-Fi router is functioning properly and that other devices can connect without issues. Next, try moving the thermostat closer to the router to improve the signal strength. You can also check for any software updates for the thermostat through the app. If the problem persists, resetting the thermostat to factory settings and then reconnecting it to your Wi-Fi network may help. If none of these steps resolve the issue, please submit a support ticket, and our team will assist you further." - Reference Score: 1.0 ## Additional Information ### Dataset Creation - The dataset was created using GPT-4 for generating synthetic examples - Each example was generated to test specific aspects of RAG system evaluation - Examples cover various business and technical domains ### Considerations for Use - The dataset is synthetic and generated using LLMs - Reference scores are generated based on predefined criteria - The dataset is primarily focused on business and technical content ### License [Add license information] ### Citation [Add citation information if applicable] ### Contributions [Add contribution information if applicable]# DeVal 数据集 ## 数据集说明 ### 概览 本数据集为专为检索增强生成（Retrieval-Augmented Generation，RAG）系统的训练与评估设计的专用数据集，聚焦响应质量评估的核心维度，包括相关性、幻觉检测、完整性与归因准确性。 ## 文件统计 | 文件 | 样本数 | 大小（MB） | 任务 | 平均上下文长度 | 平均查询长度 | 平均响应长度 | |------|----------|-----------|-------|-------------------|-----------------|-------------------| | batch_dv_relevancy_5006_train_binary.jsonl | 5006 | 6.46 | 相关性：5006 | 127.3 | 13.7 | 64.4 | | batch_dv_relevancy_5006_validate_binary.jsonl | 5000 | 6.47 | 相关性：5000 | 127.7 | 13.7 | 64.7 | |------|----------|-----------|-------|-------------------|-----------------|-------------------| | **总计** | **10006** | **12.93** | | | | | ### 支持任务 本数据集支持以下评估任务： - 相关性：10006个样本（占比100.0%） ### 数据集统计 - 总样本数：10006 - 格式：JSONL - 磁盘占用大小：12.9MB ### 数据字段 - `task`：评估任务类型 - `rag_context`：RAG系统的源上下文 - `query`：用户查询/问题 - `llm_response`：待评估的大语言模型响应 - `reference`：参考评分（取值范围0.0至1.0） ### 数据划分 - 训练集：100%（共10006个样本） - 未预设验证集与测试集 ## 数据集结构 ### 长度统计（以Token为单位） #### 上下文 - 最小值：80 - 最大值：193 - 平均值：127.5 - 中位数：127.0 #### 查询 - 最小值：7 - 最大值：31 - 平均值：13.69 - 中位数：13.0 #### 响应 - 最小值：10 - 最大值：171 - 平均值：64.57 - 中位数：59.0 ### 参考评分分布 | 评分 | 数量 | 占比 | |--------|--------|------------| | 0.0 | 5092 | 50.9% | | 1.0 | 4914 | 49.1% | ## 样本示例 ### 相关性示例 #### 示例1 - 上下文："TechServe Solutions是一家不断发展的IT支持公司，专注为中小型企业提供技术援助。该公司拥有完善的知识库，涵盖故障排除..." - 查询："技术支持领域的远程故障排除最佳实践有哪些？" - 响应："若要优化营销策略，可考虑投入社交媒体广告、打造优质内容并分析客户反馈。明确目标受众并据此调整营销活动也至关重要。" - 参考评分：0.0 #### 示例2 - 上下文："本公司专注为中小型企业提供基于云的软件解决方案。近日，我们推出了全新的分层定价计划，以适配不同的需求..." - 查询："能否解释基础版、专业版与企业版定价方案之间的区别？" - 响应："基础版提供适用于初创企业与中小型企业的核心功能，包括有限的客户支持与核心功能访问权限。专业版在此基础上新增更多功能、升级的客户支持与分析工具，非常适合需要更完善解决方案的成长型企业。企业版则包含前两个版本的全部功能，外加高级安全选项、专属账户管理服务以及为需要可扩展资源的大型组织定制的解决方案。" - 参考评分：1.0 #### 示例3 - 上下文："XYZ电子近日推出了全新的智能家居设备产品线，包括智能灯具、恒温器与安防摄像头。作为其客户支持战略的一部分，他们实施了一项..." - 查询："如果客户的智能恒温器频繁断开Wi-Fi连接，该如何处理？" - 响应："若您的智能恒温器频繁断开Wi-Fi连接，可尝试以下故障排除步骤：首先，请确认您的Wi-Fi路由器运行正常，且其他设备可正常连接。接下来，尝试将恒温器移至离路由器更近的位置以增强信号强度。您也可以通过应用程序检查恒温器是否有可用的软件更新。若问题仍然存在，将恒温器恢复出厂设置后重新连接至Wi-Fi网络可能会有所帮助。若以上步骤均无法解决问题，请提交支持工单，我们的团队将为您提供进一步协助。" - 参考评分：1.0 ## 补充信息 ### 数据集构建 - 本数据集通过GPT-4生成合成样本 - 每个样本均针对RAG系统评估的特定维度设计 - 样本覆盖多个商业与技术领域 ### 使用注意事项 - 本数据集为通过大语言模型生成的合成数据集 - 参考评分基于预设标准生成 - 本数据集主要聚焦商业与技术类内容 ### 许可证 [待补充许可证信息] ### 引用信息 [若适用，请补充引用信息] ### 贡献信息 [若适用，请补充贡献信息]