cogym-collabskill-trajectories

# Dataset Card for CollabSkill Trajectories ## Dataset Description - **Paper:** [CollabSkill: Evaluating Human-Agent Collaboration on Real-World Tasks](https://arxiv.org/abs/2606.09833) - **Project website:** [cogym.saltlab.stanford.edu](https://cogym.saltlab.stanford.edu) - **License:** [Creative Commons Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/) - **Language:** Primarily English ### Dataset Summary CollabSkill is a framework for studying how real human workers collaborate with AI agents on occupational tasks. Participants were matched to tasks based on their occupational backgrounds and paired with one of five AI agents. The release contains interaction logs, participant and session metadata, task rubrics, automated grading results, and Bayesian CollabSkill ratings for humans and agents. The tasks require realistic work products such as spreadsheets, documents, presentations, PDFs, and other deliverables. They originate from GDPVal, APEX, and APEX-Agents and cover 10 O*NET occupational sectors. See the paper for the complete collection and evaluation methodology. ## Dataset Structure ```text release/ ├── README.md ├── interaction_logs/ │ └── <session_id>/ │ └── extracted_log.jsonl └── leaderboard_data/ ├── participants.json ├── ratings.json ├── scores.json ├── sessions.json └── tasks.json ``` ### Interaction Logs Each folder under `interaction_logs/` is named with a `session_id`. A standard `extracted_log.jsonl` contains one JSON object per line with the following fields: - `session_id` (`string`): Session UUID; matches the containing directory. - `turn_id` (`integer`): Record order or turn identifier within the extracted session. - `agent` (`string`): Canonical agent name for the session. - `model` (`string`): Canonical model identifier for the session. - `role` (`string`): Record source, such as `user`, `assistant`, `tool`, or `system`. - `type` (`string`): Source-specific event type, such as `message`, `tool_call`, `tool_result`, or `status`. - `content` (`string`): Textual content or extracted event description. - `tool_name` (`string`): Tool name when applicable; otherwise an empty string. - `tool_input` (`string`): Extracted tool input when applicable; otherwise an empty string. - `is_conversational` (`boolean`): Whether the record is part of the user-facing conversation. 61 sessions do not have an extractable interaction log because the submitted log was missing, incomplete, or invalid. These files intentionally retain a distinct one-line sentinel format: ```json {"no_extracted_log_reason":"Wrong log file submission."} ``` ### Leaderboard Data - `participants.json`: participant records, including pseudonymous `user_id`, task assignments and progress, years of occupational experience, LLM-use demographics, and optional pre/post-study survey responses. - `sessions.json`: session records, exactly matching the directory names in `interaction_logs/`. - `tasks.json`: task records executed by the retained sessions, including prompts, reference-material metadata, expected deliverables, occupational metadata, and grading rubrics. - `scores.json`: automated-grader records referenced by retained sessions. Three retained sessions contain an `autograde_score_id` and embedded overall score whose detailed score record was already absent from the source `scores.json`. - `ratings.json`: Bayesian skill estimates for agents and human participants. Human `entity_id` values correspond to participant `user_id` values; agent entries use agent identifiers. The conservative CollabSkill value is `mu - 3 * sigma`. ### Joining the Files - Join an interaction-log directory to `sessions.json` with `session_id`. - Join sessions to participants with `user_id`. - Join sessions to task metadata with `task_id`. - Join human ratings to participants with `ratings.humans[].entity_id == participants[].user_id`. - Join agent ratings to session agents through the corresponding agent identifier. The participant without a human rating did not contribute a completed rated episode; therefore, 93 study participants correspond to 92 human rating entries. ### Loading the Data The release uses nested JSON and JSONL files rather than a single tabular split. The following example selects the 386 study sessions and handles both interaction-log formats: ```python import json from pathlib import Path root = Path("release") log_root = root / "interaction_logs" with (root / "leaderboard_data" / "sessions.json").open() as f: all_sessions = json.load(f) study_session_ids = {path.parent.name for path in log_root.glob("*/extracted_log.jsonl")} study_sessions = [row for row in all_sessions if row["session_id"] in study_session_ids] for path in sorted(log_root.glob("*/extracted_log.jsonl")): with path.open() as f: records = [json.loads(line) for line in f if line.strip()] if len(records) == 1 and "no_extracted_log_reason" in records[0]: reason = records[0]["no_extracted_log_reason"] continue # Process standard interaction records here. ``` ## Limitations and Intended Use - The participant sample consists of U.S.-based workers recruited through Upwork and is not representative of all workers or occupations. - Tasks cover 10 O*NET sectors and emphasize open-ended, artifact-producing work; results may not generalize to other settings. - Interaction logs come from heterogeneous agent interfaces and were normalized into a common extracted schema. - The dataset is intended for research on human-agent collaboration, interaction analysis, agent evaluation, and AI literacy. It should not be used to make consequential decisions about individual participants. ## Licensing Information This dataset release is licensed under the [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/) (`CC BY 4.0`). You may share and adapt the material for any purpose, provided that appropriate credit is given. ## Citation Information If you use this dataset, please cite: ```bibtex @inproceedings{shao2026collabskill, title = {CollabSkill: Evaluating Human-Agent Collaboration on Real-World Tasks}, author = {Shao, Yijia and Wang, Zora Zhiruo and Ahuja, Neel and Wang, Yicheng and Liu, Bowen and Yang, Diyi}, booktitle = {Third Conference on Language Modeling}, year = {2026}, url = {https://arxiv.org/abs/2606.09833} } ```