JCoLA

JCoLA（日语语言可接受性语料库）是由东京大学创建的一个包含10,020个句子的数据集，这些句子均由语言学家进行了二元可接受性判断标注。数据集中的句子主要从语言学教科书、手册和期刊文章中手动提取，并分为域内数据（86%，来自教科书和手册的相对简单的可接受性判断）和域外数据（14%，来自期刊文章的理论上有意义的可接受性判断）。域外数据根据12种语言现象进行分类。JCoLA旨在评估不同类型的日语语言模型在语法知识方面的表现，特别是在处理如论元结构等局部语法依赖和如动词一致性和非肯定词许可等长距离语法依赖时的能力。

JCoLA (Japanese Corpus of Linguistic Acceptability) is a dataset containing 10,020 sentences created by The University of Tokyo. All sentences are annotated with binary acceptability judgments by linguists. The sentences are primarily manually extracted from linguistics textbooks, handbooks and journal articles, and are divided into two subsets: in-domain data (86%), which consists of relatively simple acceptability judgments from textbooks and handbooks, and out-of-domain data (14%), which comprises theoretically meaningful acceptability judgments from journal articles. The out-of-domain subset is categorized based on 12 linguistic phenomena. JCoLA aims to evaluate the performance of different types of Japanese large language models (LLMs) on grammatical knowledge, specifically their ability to handle local syntactic dependencies such as argument structure, and long-distance syntactic dependencies such as verb agreement and non-assertive word licensing.