deepseek-v4-flash-swebench-csa-topk

# DeepSeek-V4-Flash SWE-bench CSA Top-K Traces This dataset contains compressed CSA indexer top-k traces collected from local SGLang replays of DeepSeek-V4-Flash SWE-bench agentic trajectories. It is intended for analysis of sparse-attention / CSA retrieval behavior on long SWE-bench traces. ## Dataset Summary - Source task family: SWE-bench agentic traces - Model/runtime: DeepSeek-V4-Flash replayed through SGLang - Collection mode: full trace prefill - Splits: `lite_agentic` and `verified_agentic` - Rows: 300 lite rows and 500 verified rows, 800 rows total - Size on disk: about 331 GB - Main tensor file: `indexer_topk.npz` - Main array inside NPZ: `indexer_topk` The replay uses stored multi-turn traces as prompts. The collection requests one probe token from SGLang, then stores only trace/prompt rows for analysis. Metadata records which prompt rows correspond to assistant-message spans and which correspond to system/user/tool-result spans. ## Directory Layout ```text . ├── lite_agentic_full_trace_prefill/ │ ├── README.md │ ├── manifests/ │ └── data_full_trace_prefill/lite_agentic/row_<row_index>/ │ ├── indexer_topk.npz │ ├── metadata.json │ └── response.redacted.json └── verified_agentic_full_trace_prefill/ └── data_full_trace_prefill/verified_agentic/row_<row_index>/ ├── indexer_topk.npz ├── metadata.json └── response.redacted.json ``` ## File Format Each row directory stores: - `indexer_topk.npz`: compressed NumPy archive containing `indexer_topk` - `metadata.json`: provenance, model config, token boundaries, row spans, and shape/collection details - `response.redacted.json`: redacted response metadata from the collection request `indexer_topk` is an `int32` tensor with shape: ```text [trace_prompt_tokens - 1, 21, 512] ``` Axis meanings: - axis 0: token transition row over the full replayed trace prompt - axis 1: compact CSA slot id, not the original hidden layer id - axis 2: top-k rank from the DeepSeek-V4 CSA indexer The CSA slot mapping is recorded in each `metadata.json`. For this collection, slots map to even hidden layers from 2 through 42. ## Loading Example ```python import json import numpy as np from pathlib import Path row = Path("lite_agentic_full_trace_prefill/data_full_trace_prefill/lite_agentic/row_0000") with np.load(row / "indexer_topk.npz") as archive: indexer_topk = archive["indexer_topk"] metadata = json.loads((row / "metadata.json").read_text()) print(indexer_topk.shape) print(metadata["source"]["instance_id"]) print(metadata["token_boundaries"]["assistant_decode_equivalent_row_count"]) ``` ## Provenance The source trace package describes DeepSeek V4 Flash agentic SWE-bench trajectories collected with EvalScope and the DeepSeek API. This dataset does not re-run SWE-bench containers; it replays the stored message traces against a local SGLang DeepSeek-V4-Flash service and captures returned `sglext.indexer_topk` data. Original source splits reported by the local trace package: - `lite_agentic`: 300 rows, score summary `182 / 300` - `verified_agentic`: 500 rows, score summary `354 / 500` ## Notes and Limitations - The data reflects local SGLang replay, not the original DeepSeek API decode. - Full-trace prefill avoids natural decode divergence by using the stored trace as prompt text. - Assistant spans in `metadata.json` are decode-equivalent spans reconstructed from the stored trace, not newly generated assistant tokens. - Top-k ids are DeepSeek-V4 indexer raw ids; they are not FlashMLA page ids. - Files are large. Prefer streaming row-by-row rather than loading the entire dataset at once.