ali-vosoughi/oscar-dataset

--- pretty_name: OSCaR language: - en license: other task_categories: - image-to-text task_ids: - image-captioning size_categories: - 10K<n<100K --- # OSCaR OSCaR is the public dataset release for the NAACL 2024 paper _Object State Captioning and State Change Representation_. This release packages the preserved OSCaR image assets, fine-tuning manifests, benchmark split metadata, and state-caption sidecars used around the LLaVA-based training and evaluation workflow published in the [OSCaR GitHub repository](https://github.com/nguyennm1024/OSCaR). ## Release Summary - Paper-reported scale: **14,084** annotated segments across EPIC-KITCHENS and Ego4D. - Public raw asset tree in this release: **7,742** clip directories under `data/object-state-data`. - Full preserved image-caption mapping: **30,308** rows across **7,577** clips. - LLaVA fine-tuning manifest: **28,308** image-level conversations across **7,077** clips. - Human-verified EPIC benchmark split: **2,000** rows / **500** clips / 4 caption slots. - Sidecar annotations included: **7,586** state-change JSON files, **2,244** QA JSON files, **3,142** conversation JSON files. - Open-world evaluation metadata included: **356** Ego4D records and **344** EPIC-KITCHENS records. ## What Is Included - `data/object-state-data/`: preserved OSCaR frame directories and `state_change.jpg` composites. - `manifests/llava_data.json`: OSCaR fine-tuning manifest used for adapter training. - `splits/data_mapping_final_EK_test.csv`: held-out human-verified EPIC benchmark split. - `metadata/data_mapping_final.csv`: full preserved image-to-caption mapping. - `metadata/video-object.csv`: narration-to-object/action table. - `metadata/ego4d_data.csv`: preserved Ego4D action/object metadata. - `annotations/state-change-json/`: state caption JSON sidecars. - `annotations/question-answers-clean/`: optional QA sidecars. - `annotations/conversation-clean/`: optional conversation sidecars. - `eval/openworld.json` and `eval/openworld-epic.json`: open-world evaluation prompts/metadata. ## Directory Layout ```text oscar-dataset/ data/object-state-data/ manifests/llava_data.json splits/data_mapping_final_EK_test.csv metadata/data_mapping_final.csv metadata/segment_index.csv metadata/release_summary.json annotations/state-change-json/ annotations/question-answers-clean/ annotations/conversation-clean/ eval/openworld.json eval/openworld-epic.json ``` ## Important Notes - The paper reports 14,084 annotated segments, but the preserved public asset tree in this release contains 7,742 clip directories. The released metadata keeps both the paper-scale claim and the preserved local archive counts explicit. - `metadata/segment_index.csv` is the normalized release table generated from the preserved asset tree, the full mapping CSV, the fine-tuning manifest, and the benchmark split. - Some open-world evaluation JSON records still reference original local EPIC or Ego4D frame roots. Those records are included for provenance and regeneration, not as a promise that every referenced raw frame path is redistributed here. ## Usage With OSCaR Code The public code release expects a workspace like: ```text workspace/ OSCaR/ oscar-dataset/ ``` Then run, for example: ```bash DATASET_ROOT=../oscar-dataset \ bash scripts/train/finetune_v1_5_13b_oscar_lora.sh ``` ## Provenance - Source corpora: EPIC-KITCHENS and Ego4D, as described in the paper. - Public code: `nguyennm1024/OSCaR` - Public model namespace: `ali-vosoughi` - Dataset repo: `ali-vosoughi/oscar-dataset` ## Citation ```bibtex @inproceedings{nguyen2024oscar, title={OSCaR: Object State Captioning and State Change Representation}, author={Nguyen, Nguyen and Bi, Jing and Vosoughi, Ali and Tian, Yapeng and Fazli, Pooyan and Xu, Chenliang}, booktitle={North American Chapter of the Association for Computational Linguistics (NAACL)}, year={2024} } ```--- pretty_name: OSCaR language: - 英语 license: 其他 task_categories: - 图像到文本（image-to-text） task_ids: - 图像字幕生成（image-captioning） size_categories: - 10000 < 样本量 < 100000 --- # OSCaR数据集 OSCaR是发表于北美计算语言学协会（Association for Computational Linguistics, NAACL）2024年会的论文《对象状态字幕与状态变化表征（Object State Captioning and State Change Representation）》的公开数据集发布版本。 本发布包整合了基于LLaVA的训练与评估流程中所需的经留存处理的OSCaR图像资源、微调配置清单、基准划分元数据以及状态字幕附属文件，相关完整内容可查阅[OSCaR GitHub仓库](https://github.com/nguyennm1024/OSCaR)。 ## 发布概况 - 论文报告的数据集规模：**14,084** 个标注片段，覆盖EPIC-KITCHENS与Ego4D两大数据集。 - 本发布包中的公开原始资源目录：`data/object-state-data` 路径下共计 **7,742** 个剪辑目录。 - 完整留存的图像-字幕映射关系：共 **30,308** 条数据，覆盖 **7,577** 个剪辑。 - LLaVA微调配置清单：**28,308** 条图像级对话数据，覆盖 **7,077** 个剪辑。 - 人工验证的EPIC基准划分：**2,000** 条数据 / **500** 个剪辑 / 4个字幕槽位。 - 包含的附属标注文件：**7,586** 个状态变化JSON文件、**2,244** 个问答（QA）JSON文件、**3,142** 个对话JSON文件。 - 包含的开放域评估元数据：**356** 条Ego4D记录与 **344** 条EPIC-KITCHENS记录。 ## 包含内容 - `data/object-state-data/`：经留存处理的OSCaR帧目录与`state_change.jpg`合成图像文件。 - `manifests/llava_data.json`：用于适配器训练的OSCaR微调配置清单。 - `splits/data_mapping_final_EK_test.csv`：预留的经人工验证的EPIC基准划分数据集。 - `metadata/data_mapping_final.csv`：完整的留存图像-字幕映射表。 - `metadata/video-object.csv`：叙述-对象/动作对应表。 - `metadata/ego4d_data.csv`：经留存处理的Ego4D动作/对象元数据。 - `annotations/state-change-json/`：状态字幕JSON附属文件。 - `annotations/question-answers-clean/`：可选的问答附属文件。 - `annotations/conversation-clean/`：可选的对话附属文件。 - `eval/openworld.json` 与 `eval/openworld-epic.json`：开放域评估提示词与元数据文件。 ## 目录结构 text oscar-dataset/ data/object-state-data/ manifests/llava_data.json splits/data_mapping_final_EK_test.csv metadata/data_mapping_final.csv metadata/segment_index.csv metadata/release_summary.json annotations/state-change-json/ annotations/question-answers-clean/ annotations/conversation-clean/ eval/openworld.json eval/openworld-epic.json ## 重要说明 - 论文报告共包含14,084个标注片段，但本发布包中的公开留存资源目录仅包含7,742个剪辑目录。本次发布的元数据会明确区分论文报告的数据集规模与本地留存归档的实际资源数量。 - `metadata/segment_index.csv` 是基于留存资源目录、完整映射表、微调配置清单以及基准划分数据集生成的标准化发布表格。 - 部分开放域评估JSON记录仍引用了原始本地EPIC-KITCHENS或Ego4D帧文件根路径，此类记录仅用于溯源与复现流程，并非承诺所有引用的原始帧文件均已在此发布包中重新分发。 ## 配合OSCaR代码库使用 公开的代码发布版本预期的工作空间结构如下： text workspace/ OSCaR/ oscar-dataset/ 例如执行以下命令启动训练： bash DATASET_ROOT=../oscar-dataset bash scripts/train/finetune_v1_5_13b_oscar_lora.sh ## 溯源信息 - 源语料库：如论文所述，为EPIC-KITCHENS与Ego4D两大数据集。 - 公开代码仓库：`nguyennm1024/OSCaR` - 公开模型命名空间：`ali-vosoughi` - 数据集仓库：`ali-vosoughi/oscar-dataset` ## 引用格式 bibtex @inproceedings{nguyen2024oscar, title={OSCaR: 对象状态字幕与状态变化表征（Object State Captioning and State Change Representation）}, author={Nguyen, Nguyen and Bi, Jing and Vosoughi, Ali and Tian, Yapeng and Fazli, Pooyan and Xu, Chenliang}, booktitle={North American Chapter of the Association for Computational Linguistics (NAACL)}, year={2024} }