Salesforce/cota-mantis

TACO数据集是一个包含293K个合成思维和动作链（CoTA）的数据集，这些链由GPT-4o生成，涉及15种不同的动作，如OCR、深度估计、计算等。该数据集的主要用途是微调多模态语言模型，以产生思维和动作链来回答复杂的视觉问题。数据集来源于Cauldron和Mantis-Instruct，这些数据是从COCO、AOKVQA、ScienceQA、Visual Genome等现有数据集中收集的。数据集的局限性包括GPT-4o的偏见和动作的局限性，主要涵盖视觉中心工具和一些通用工具。

The TACO dataset is a collection of 293K synthetic chains of thoughts and actions (CoTA) generated by GPT-4o, involving 15 different actions such as OCR, depth estimation, calculation, etc. The primary use of this dataset is to fine-tune multi-modal language models to produce chains of thoughts and actions to answer complex visual questions. The dataset is sourced from Cauldron and Mantis-Instruct, which are collected from various existing datasets including COCO, AOKVQA, ScienceQA, Visual Genome, etc. The limitations of the dataset include biases inherited from GPT-4o and the limited scope of actions, mostly covering vision-centric tools and some generic tools.