RWSD (The Winograd Schema Challenge (Russian))

Winograd 模式是一对只有一个或两个单词不同的句子，并且包含在两个句子中以相反方式解决的歧义，并且需要使用世界知识和推理来解决它。该模式的名称取自 Terry Winograd 的一个著名示例。然后，按照图灵测试的思路，该集合将作为 AI 程序的挑战提出。挑战的优势在于它是明确的，因为每个模式的答案都是二元选择；生动，因为对于非专家来说，显然无法获得正确答案的程序在理解上存在严重差距；并且困难，因为它远远超出了当前的技术水平。任务类型逻辑和推理，世界知识。二进制分类：真/假示例 {“文本”：“Кубок не помещается в коричневый чемодан, потому что он слишком большой。” "label": false, "idx": 5, "target": { "span1_text": "чемодан", "span2_text": "он слишком большой", "span1_index": 5, "span2_index": 8 }, } 如何我们收集数据了吗？所有文本示例都是手动收集的，将原始 Winograd 数据集翻译和改编为俄语。在 Yandex.Toloka 上进行了人工评估。

Winograd Schemas are pairs of sentences that differ by only one or two words, and contain an ambiguity that is resolved in opposite ways in the two sentences, requiring world knowledge and reasoning to resolve. The name of the schema is derived from a famous example by Terry Winograd. Then, following the spirit of the Turing Test, this collection is presented as a challenge for AI programs. The advantages of this challenge are as follows: it is explicit, as each schema's answer is a binary choice; it is vivid, as a program that fails to obtain the correct answer clearly shows a serious gap in understanding for non-experts; and it is difficult, as it far exceeds the current state of the art. Task types: logical reasoning, world knowledge. Binary classification: True/False.

Example: {"text": "Кубок не помещается в коричневый чемодан, потому что он слишком большой。", "label": false, "idx": 5, "target": { "span1_text": "чемодан", "span2_text": "он слишком большой", "span1_index": 5, "span2_index": 8 }, }

How did we collect the data? All text examples were manually collected by translating and adapting the original Winograd dataset into Russian. Human evaluation was conducted on Yandex.Toloka.

搜集汇总

背景与挑战

背景概述

RWSD是俄语版的Winograd Schema Challenge数据集，旨在通过二元分类任务评估AI模型的世界知识和推理能力，要求模型解决仅有一两个单词差异的歧义句子。该数据集基于原始英文版本手动翻译和改编为俄语，并经过了人工评估验证。