deepseek-v4-flash-swebench-replay

# deepseek-v4-flash-swebench-replay ## 中文 这是一个 **DeepSeek V4 Flash 在 SWE-bench 上的 agentic replay 数据集仓库**。 它的目标是让使用者**不需要部署 SWE-bench**，也不需要复现 Docker/benchmark 环境，就可以直接查看和重放模型的多轮推理与工具调用轨迹。 ### 当前包含的数据 - `verified_agentic` - `lite_agentic` ### 当前不包含的数据 - 单轮 single-turn trace - `verified_mini_agentic`（当前本地仅完成 31/50，因此不纳入首版） ### 分数汇总 - `verified_agentic`: `354 / 500`, `Acc/Pass@1 = 70.8` - `lite_agentic`: `182 / 300`, `Acc/Pass@1 = 60.67` ### 数据来源 这些轨迹由 EvalScope + DeepSeek API 收集，使用的是 EvalScope 中提供的 official SWE-bench agentic benchmark 数据集与执行链路。 运行时环境基于 rootless Docker 和 benchmark 官方容器镜像。 相关参考链接： - EvalScope SWE-bench 文档： https://evalscope.readthedocs.io/zh-cn/latest/third_party/swe_bench.html - EvalScope SWE-bench Verified Agentic： https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/swe_bench_verified_agentic.html - EvalScope SWE-bench Lite Agentic： https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/swe_bench_lite_agentic.html ### 仓库结构 ```text data/ verified_agentic/ replay_dataset.jsonl samples.jsonl predictions.jsonl lite_agentic/ replay_dataset.jsonl samples.jsonl predictions.jsonl manifests/ release_manifest.json verified_agentic_manifest.json lite_agentic_manifest.json scores/ overview.json verified_agentic.json lite_agentic.json examples/ replay_verified_agentic_sample.json replay_lite_agentic_sample.json scripts/ replay_openai_compatible.py inspect_trace.py ``` ### 如何读取数据 #### `samples.jsonl` 每一行是一个完整样本，包含： - `instance_id` - `messages` - `agent_trace` - `trace_status` - `eval_status` - `patch` - `final_content` - `reasoning_content` - `tool_calls` #### `replay_dataset.jsonl` 这是重放入口。 它保留多轮消息顺序，可直接用于 OpenAI-compatible endpoint 重放。 #### `predictions.jsonl` 保留最终 patch 结果，适合做外部评测、快速检查或与原 benchmark 结果对照。 ### 如何重放 ```bash python scripts/replay_openai_compatible.py \ --dataset data/verified_agentic/replay_dataset.jsonl \ --api-base https://api.deepseek.com \ --api-key-env DEEPSEEK_API_KEY \ --model deepseek-v4-flash \ --limit 1 ``` ### 如何查看某一条轨迹 ```bash python scripts/inspect_trace.py \ --samples data/verified_agentic/samples.jsonl \ --instance-id astropy__astropy-12907 ``` ### 局限性 - 这是采样得到的运行轨迹，不是官方 leaderboard 的原始发布物。 - 当前首版只发布 `verified_agentic` 和 `lite_agentic`。 - `verified_mini_agentic` 未纳入首版，以保持数据完整性和口径稳定。 ### 致谢 感谢 [算苗](https://www.sunmmio.com/) 提供本次采集所需的服务器支持。 ## English This repository publishes **agentic SWE-bench replay traces for DeepSeek V4 Flash**. The goal is to let users inspect and replay the model's multi-turn reasoning and tool-use behavior **without setting up SWE-bench locally**. ### Included splits - `verified_agentic` - `lite_agentic` ### Excluded from v1 - single-turn traces - `verified_mini_agentic` (local coverage is only 31/50, so it is excluded from the first public release) ### Score summary - `verified_agentic`: `354 / 500`, `Acc/Pass@1 = 70.8` - `lite_agentic`: `182 / 300`, `Acc/Pass@1 = 60.67` ### Data provenance The traces were collected with EvalScope + the DeepSeek API, using the official SWE-bench agentic benchmarks exposed by EvalScope. Execution used rootless Docker and official benchmark containers. Reference links: - EvalScope SWE-bench docs: https://evalscope.readthedocs.io/zh-cn/latest/third_party/swe_bench.html - EvalScope SWE-bench Verified Agentic: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/swe_bench_verified_agentic.html - EvalScope SWE-bench Lite Agentic: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/swe_bench_lite_agentic.html ### How to use the data - `data/<split>/samples.jsonl`: full trace rows with messages, agent_trace, trace/eval status, patch and final content - `data/<split>/replay_dataset.jsonl`: replay-oriented multi-turn message rows - `data/<split>/predictions.jsonl`: final patch predictions keyed by instance ### Replay example ```bash python scripts/replay_openai_compatible.py \ --dataset data/verified_agentic/replay_dataset.jsonl \ --api-base https://api.deepseek.com \ --api-key-env DEEPSEEK_API_KEY \ --model deepseek-v4-flash \ --limit 1 ``` ### Inspect example ```bash python scripts/inspect_trace.py \ --samples data/verified_agentic/samples.jsonl \ --instance-id astropy__astropy-12907 ``` ### Acknowledgements Thanks to [Sunmmio](https://www.sunmmio.com/) for providing the server support used in this collection.