OpenGVLab/CRPE
收藏资源简介：
--- license: apache-2.0 --- # Circular-based Relation Probing Evaluation (CRPE) CRPE is a benchmark designed to quantitatively evaluate the object recognition and relation comprehension ability of models. The evaluation is formulated as single-choice questions. The benchmark consists of four splits: **Existence**, **Subject**, **Predicate**, and **Object**. The **Existence** split evaluates the object recognition ability while the remaining splits are designed to evaluate the capability of relation comprehension, focusing on probing each of the elements in the relation triplets `(subject, predicate, object)` separately. Some data examples are shown below. <img width="800" alt="image" src="https://cdn-uploads.huggingface.co/production/uploads/619507e7b74b6c591f794340/_NKaowl2OUBAjck1XCAPm.jpeg"> Additionally, to evaluate the dependency on language priors, we also include abnormal data in our evaluation. These images in these abnormal data depict relation triplets that are very rare in the real world. <img width="800" alt="image" src="https://cdn-uploads.huggingface.co/production/uploads/619507e7b74b6c591f794340/qKWw7Qb93OXClxI_VrCRk.jpeg"> For a robust evaluation, we adopt CircularEval as our evaluation strategy. Under this setting, a question is considered as correctly answered only when the model consistently predicts the correct answer in each of the N iterations, with N corresponding to the number of choices. In each iteration, a circular shift is applied to both the choices and the answer to form a new query for the model. CRPE contains the following files: - `crpe_exist.jsonl`: the evaluation data of **Existence** split. - `crpe_exist_meta.jsonl`: the evaluation data of **Existence** split without CircularEval. - `crpe_relation.jsonl`: the evaluation data of **Subject**, **Predicate**, and **Object** split. - `crpe_relation_meta.jsonl`: the evaluation data of **Subject**, **Predicate**, and **Object** split without CircularEval. **NOTE**: You should use `crpe_exist.jsonl` and `crpe_relation.jsonl` for evaluation. The evaluation script is presented [here](https://github.com/OpenGVLab/all-seeing/blob/main/all-seeing-v2/llava/eval/eval_crpe.py). See our [project](https://github.com/OpenGVLab/all-seeing/all-seeing-v2) to learn more details! # Citation If you find our work useful in your research, please consider cite: ```BibTeX @article{wang2023allseeing, title={The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World}, author={Wang, Weiyun and Shi, Min and Li, Qingyun and Wang, Wenhai and Huang, Zhenhang and Xing, Linjie and Chen, Zhe and Li, Hao and Zhu, Xizhou and Cao, Zhiguo and others}, journal={arXiv preprint arXiv:2308.01907}, year={2023} } @article{wang2024allseeing_v2, title={The All-Seeing Project V2: Towards General Relation Comprehension of the Open World}, author={Wang, Weiyun and Ren, Yiming and Luo, Haowen and Li, Tiantong and Yan, Chenxiang and Chen, Zhe and Wang, Wenhai and Li, Qingyun and Lu, Lewei and Zhu, Xizhou and others}, journal={arXiv preprint arXiv:2402.19474}, year={2024} } ```
--- license: apache-2.0 --- # 基于循环的关系探测评估基准（Circular-based Relation Probing Evaluation, CRPE） CRPE是一款专为定量评估模型的目标识别与关系理解能力设计的基准测试集，本次评估以单项选择题为形式构建。 该基准测试集包含四个划分子集：**存在性（Existence）**、**主体（Subject）**、**谓词（Predicate）**与**客体（Object）**。 其中**存在性**子集用于评估模型的目标识别能力，其余三个子集则聚焦于关系理解能力测评，分别对关系三元组`(subject, predicate, object)`中的各元素进行单独探测。部分数据示例如下： <img width="800" alt="image" src="https://cdn-uploads.huggingface.co/production/uploads/619507e7b74b6c591f794340/_NKaowl2OUBAjck1XCAPm.jpeg"> 此外，为评估模型对语言先验的依赖程度，本次测试还引入了异常数据样本。此类异常数据中的图像所描绘的关系三元组在现实世界中极为罕见。 <img width="800" alt="image" src="https://cdn-uploads.huggingface.co/production/uploads/619507e7b74b6c591f794340/qKWw7Qb93OXClxI_VrCRk.jpeg"> 为保障评估的稳健性，我们采用循环评估（CircularEval）作为评估策略。在该策略下，仅当模型在N轮迭代中均能始终如一地预测出正确答案时，才认为该问题回答正确，其中N等于选项的数量。每一轮迭代中，我们都会对选项与标准答案进行循环移位，为模型生成全新的查询任务。 CRPE包含以下文件： - `crpe_exist.jsonl`：**存在性**子集的评估数据 - `crpe_exist_meta.jsonl`：未采用循环评估策略的**存在性**子集评估数据 - `crpe_relation.jsonl`：**主体**、**谓词**与**客体**子集的评估数据 - `crpe_relation_meta.jsonl`：未采用循环评估策略的**主体**、**谓词**与**客体**子集评估数据 **注意**：评估时请使用`crpe_exist.jsonl`与`crpe_relation.jsonl`。评估脚本可参见[此处](https://github.com/OpenGVLab/all-seeing/blob/main/all-seeing-v2/llava/eval/eval_crpe.py)。 如需了解更多细节，请参阅我们的[项目页面](https://github.com/OpenGVLab/all-seeing/all-seeing-v2)！ # 引用声明 若您的研究中用到了本工作，请引用如下文献： BibTeX @article{wang2023allseeing, title={The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World}, author={Wang, Weiyun and Shi, Min and Li, Qingyun and Wang, Wenhai and Huang, Zhenhang and Xing, Linjie and Chen, Zhe and Li, Hao and Zhu, Xizhou and Cao, Zhiguo and others}, journal={arXiv preprint arXiv:2308.01907}, year={2023} } @article{wang2024allseeing_v2, title={The All-Seeing Project V2: Towards General Relation Comprehension of the Open World}, author={Wang, Weiyun and Ren, Yiming and Luo, Haowen and Li, Tiantong and Yan, Chenxiang and Chen, Zhe and Wang, Wenhai and Li, Qingyun and Lu, Lewei and Zhu, Xizhou and others}, journal={arXiv preprint arXiv:2402.19474}, year={2024} }
数据集概述
数据集名称
- Circular-based Relation Probing Evaluation (CRPE)
数据集目的
- 用于量化评估模型的对象识别和关系理解能力。
数据集结构
- 四个分割：
- Existence：评估对象识别能力。
- Subject, Predicate, Object：评估关系理解能力，分别针对关系三元组中的元素进行探测。
数据集内容
- 文件列表：
crpe_exist.jsonl：Existence分割的评估数据。crpe_exist_meta.jsonl：Existence分割的评估数据，不含CircularEval。crpe_relation.jsonl：Subject, Predicate, Object分割的评估数据。crpe_relation_meta.jsonl：Subject, Predicate, Object分割的评估数据，不含CircularEval。
评估方法
- CircularEval：通过多次迭代和循环移位来评估模型的一致性。
异常数据
- 包含描绘现实中罕见关系三元组的图像，用于评估模型对语言先验的依赖。
使用建议
- 推荐使用
crpe_exist.jsonl和crpe_relation.jsonl进行评估。
