codex_swebenchpro_traces_Otel

# Codex SWE-Bench Pro — OTel Traces OpenTelemetry-formatted LLM traces derived from [Inferact/codex_swebenchpro_traces](https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces), a collection of agentic Codex runs on the [SWE-bench Pro](https://www.swebench.com/) software-engineering benchmark. ## Overview Each row in the source dataset is a full multi-turn agent conversation where a Codex agent resolves a real GitHub issue. This dataset re-represents those conversations as **OpenTelemetry GenAI spans**, one span per LLM call, using cumulative message history so that each span captures exactly what the model received and produced at that step. **Note on assistant message content:** The source dataset redacts all model outputs — every assistant message is replaced with lorem ipsum placeholder text. User-side messages (tool outputs, shell command results, file contents) are real. This dataset is therefore useful for studying LLM input structure and context growth patterns, but not model output behavior. **Note on model identity:** The source dataset does not expose a model identifier. `gen_ai.request.model` and `gen_ai.response.model` are set to `"unknown"`. ## License Released under [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/). ## Conversion Logic For a conversation with turns `[user₁, assistant₁, user₂, assistant₂, …]`: | Span | `gen_ai.input.messages` | `gen_ai.output.messages` | |------|------------------------|--------------------------| | 1 | `[user₁]` | `[assistant₁]` | | 2 | `[user₁, assistant₁, user₂]` | `[assistant₂]` | | 3 | `[user₁, assistant₁, user₂, assistant₂, user₃]` | `[assistant₃]` | Timestamps are synthetic: spans within a trace are spaced with random 1–10 second delays (no real wall-clock timing data was available in the source). ## Dataset Statistics | Metric | Value | |--------|-------| | Total traces | 610 | | Total spans | 20,230 | | Mean spans / trace | 33.2 | | Median spans / trace | 30 | | Min / max spans / trace | 6 / 100 | ## Dataset Structure Each file is a single-line JSONL object (one trace per line): ```json { "trace_id": "<32-char hex>", "span_count": 11, "collected_at": "<ISO timestamp>", "spans": [...] } ``` Each span: ```json { "trace_id": "...", "span_id": "...", "parent_span_id": null, "name": "chat unknown", "kind": "SPAN_KIND_CLIENT", "start_time": "2026-05-24T07:52:24.216485", "end_time": "2026-05-24T07:52:24.216485", "attributes": { "gen_ai.operation.name": "chat", "gen_ai.request.model": "unknown", "gen_ai.response.model": "unknown", "gen_ai.input.messages": "<JSON-encoded message array>", "gen_ai.output.messages": "<JSON-encoded message array>", "gen_ai.tool.definitions": "[]" }, "resource_attributes": { "telemetry.sdk.language": "python", "telemetry.sdk.name": "codex", "telemetry.sdk.version": "1.0.0", "service.name": "codex", "service.version": "1.0.0" }, "status": { "code": 1, "message": "" } } ``` Note: `gen_ai.input.messages` and `gen_ai.output.messages` are **JSON-encoded strings** (not parsed arrays). Each message follows the OTel GenAI format: ```json { "role": "user" | "assistant", "parts": [{ "type": "text", "content": "..." }] } ``` ## Usage ```python import json from datasets import load_dataset ds = load_dataset("json", data_files="*.jsonl", split="train") # Each row is one trace trace = ds[0] print(f"{trace['span_count']} spans in this trace") # Iterate spans for span in trace["spans"]: attrs = span["attributes"] input_msgs = json.loads(attrs["gen_ai.input.messages"]) output_msgs = json.loads(attrs["gen_ai.output.messages"]) print(f"span {span['span_id']} — {len(input_msgs)} input messages") for msg in output_msgs: for part in msg.get("parts", []): print(f" [{msg['role']}] {part['content'][:100]}") ``` ## Source Dataset - **HF repo:** [Inferact/codex_swebenchpro_traces](https://huggingface.co/datasets/Inferact/codex_swebenchpro_traces) - **Task:** SWE-bench Pro — resolving GitHub issues across 11 open-source Python repositories - **Agent:** Codex (OpenAI) - **Original size:** 610 successful trials out of 731 total (~54% pass rate)