stackexchange-superuser-sandboxes-verified
收藏资源简介：
该数据集是一个包含10,000个任务的Harbor任务二值数据集，专为智能代理和强化学习场景设计。数据来源于Super User StackExchange网站上关于Linux/Windows桌面系统、硬件配置、存储管理和终端用户系统管理的计算机使用问题。每个任务包含path（路径字符串）和task_binary（gzip压缩包）两个核心字段，遵循Harbor/TaskTrove任务规范。数据集基于DCAgent/stackexchange-superuser-sandboxes-skywork-response重新打包，完整保留了原始任务指令文件（instruction.md）和统一的Ubuntu 24.04沙盒环境。关键特性在于集成了LLM-judge验证器系统（移植自OpenThoughts-Agent框架），替代了原有的奖励模型验证器。验证器在运行时读取代理生成的响应文件（/app/response.txt）和包含任务指令及评分标准的验证数据（/tests/verifier_data.json），通过litellm接口调用LLM法官（默认使用GPT-4o-mini）进行多维度评估，将0.0-1.0的奖励分数写入日志文件。验证过程需要配置OPENAI_API_KEY环境变量（通过task.toml文件传递），并可选择性地覆盖法官模型。评分标准全面覆盖技术响应的正确性、解决方案的完整性、操作的安全性和回答的相关性四个维度，专门用于评估现实世界桌面及系统管理场景的请求处理质量。该数据集适用于文本生成、智能代理训练、强化学习评估和LLM法官系统开发等任务。
This Harbor task binary dataset comprises 10,000 tasks, tailored for AI Agent and reinforcement learning scenarios. The dataset is sourced from computer usage questions related to Linux/Windows desktop systems, hardware configuration, storage management, and end-user system administration on the Super User StackExchange website. Each task includes two core fields: `path` (path string) and `task_binary` (gzip-compressed archive), adhering to the Harbor/TaskTrove task specification. The dataset is repackaged based on the DCAgent/stackexchange-superuser-sandboxes-skywork-response repository, fully retaining the original task instruction file (instruction.md) and the standardized Ubuntu 24.04 sandbox environment. Its key feature is the integration of the LLM-judge validator system, which is ported from the OpenThoughts-Agent framework, replacing the original reward model validator. During runtime, the validator reads the agent-generated response file (/app/response.txt) and the verification data containing task instructions and scoring criteria (/tests/verifier_data.json), invokes the LLM judge via the litellm interface (GPT-4o-mini is used by default) for multi-dimensional evaluation, and writes the reward score ranging from 0.0 to 1.0 into the log file. The validation process requires configuring the OPENAI_API_KEY environment variable (passed via the task.toml file), and the judge model can be optionally overridden. The scoring criteria comprehensively cover four dimensions: correctness of technical responses, completeness of solutions, operational safety, and answer relevance, and are specifically designed to evaluate the quality of request processing in real-world desktop and system administration scenarios. This dataset is applicable to tasks such as text generation, AI Agent training, reinforcement learning evaluation, and LLM judge system development.
数据集概述：laion/stackexchange-superuser-sandboxes-verified
- 许可证：Apache-2.0
- 任务类别：文本生成（text-generation）
- 标签：agent, harbor, reinforcement-learning, llm-judge
数据集内容
- 规模：包含10,000个任务（binary task data）。
- 数据列：
path（字符串类型）和task_binary（gzip tar格式）。 - 任务来源：基于 StackExchange 平台 Super User 板块的问题，涵盖 Linux/Windows 桌面操作、硬件、存储、终端用户系统管理等真实计算机使用场景。
- 任务架构：遵循标准的 Harbor / TaskTrove 任务模式。
- 数据来源与处理：从
DCAgent/stackexchange-superuser-sandboxes-skywork-response重新打包而来；保留了原始的instruction.md文件和共享的ubuntu:24.04环境。
验证机制
- 验证器：每个任务沙箱自带一个基于 LLM（大型语言模型）的评判验证器，源自 OpenThoughts-Agent 的
data.nemotron_gym框架（具体文件为verifiers/llm_judge.py），替代了原始的 Skywork 奖励模型验证器。 - 验证流程：在验证时，脚本
tests/test_state.py会读取 Agent 生成的/app/response.txt以及/tests/verifier_data.json（包含任务指令和适配的评分准则）。随后通过litellm调用 LLM 评判模型（默认为openai/gpt-4o-mini），并将评分（0.0 到 1.0 之间的奖励值）写入/logs/verifier/reward.txt。 - 运行要求：在实验阶段需要设置
OPENAI_API_KEY环境变量（通过task.toml的[verifier].env配置传播给验证容器），并可通过JUDGE_MODEL参数覆盖默认评判模型。
评分准则
- 针对真实世界的桌面/系统管理请求，从 正确性、完整性、安全性、相关性 四个维度进行分级评分。
