projecte-aina/vinclat

--- pretty_name: Vinclat language: - ca license: cc-by-4.0 task_categories: - text-generation - question-answering size_categories: - n<1K annotations_creators: - expert-generated language_creators: - expert-generated tags: - benchmark - evaluation - catalan - catalonia - game - riddles dataset_info: dataset_size: 1000 splits: - name: train num_examples: 1000 features: - name: id dtype: int32 - name: hint_1 dtype: string - name: hint_2 dtype: string - name: hint_3 dtype: string - name: hint_4 dtype: string - name: keyword_1 dtype: string - name: keyword_2 dtype: string - name: keyword_3 dtype: string - name: keyword_4 dtype: string - name: solution_words_len sequence: int64 - name: solution_pattern dtype: string --- # Dataset Card for Vinclat **Vinclat** is a Catalan-language dataset for multi-step problem solving. It employs a game-based structure to evaluate both the reasoning capabilities and cultural knowledge of large language models. - **Language(s):** Catalan - **Paper:** [ACL Anthology](https://aclanthology.org/2026.mme-main.4/) - **Leaderboard:** [HF Space](https://huggingface.co/spaces/projecte-aina/vinclat-leaderboard) - **Code:** [GitHub](https://github.com/mapama247/vinclat_eval) - **License:** [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/deed.en) - **Funded by:** [Projecte Aina](https://projecteaina.cat/en/) - **Curated by:** Barcelona Supercomputing Center - **Shared by:** Barcelona Supercomputing Center ## Dataset Description **Vinclat** is a Catalan-language evaluation dataset designed to assess the reasoning capabilities and cultural knowledge of large language models (LLMs). It contains 1,000 instance meticulously crafted and reviewed by human annotators. Each instance follows a game-based structure in which models must solve a complex riddle through a multi-step reasoning process. Given four independent clues, models are supposed to infer intermediate concepts which, despite being seemingly unrelated, can be creatively connected to reach a final solution. Successfully solving the tasks requires a blend of reasoning strategies and linguistic understanding, combining logical inference with knowledge of Catalan language and culture. To preserve the long-term validity of the benchmark, the dataset release does not provide the final solutions. Instead, an external leaderboard is maintained by the authors, where state-of-the-art models are evaluated and added over time. This approach helps prevent benchmark contamination and ensures that the dataset remains a reliable evaluation resource. Researchers who would like their models to be included in the leaderboard are encouraged to contact the authors. ## Dataset Structure ### Data Fields - `id` (int): Unique ID assigned to each instance. - `hint_1` (str): First hint. - `hint_2` (str): Second hint. - `hint_3` (str): Third hint. - `hint_4` (str): Fourth hint. - `keyword_1` (str): Concept associated to the first hint. - `keyword_2` (str): Concept associated to the second hint. - `keyword_3` (str): Concept associated to the third hint. - `keyword_4` (str): Concept associated to the fourth hint. - `solution_words_len` (list[int]): A list indicating the number of letters in each word of the solution, in order. - `solution_pattern` (str): A visual pattern of the solution using underscores to represent each letter, with spaces separating words. ### Data instances ``` { 'id': 1, 'hint_1': 'El sufix de la majoria d’adverbis', 'hint_2': 'La platja més surfista de Cadis', 'hint_3': 'De Shock o de parella', 'hint_4': 'La filla de la paciència', 'keyword_1': 'Ment', 'keyword_2': 'Tarifa', 'keyword_3': 'Teràpia', 'keyword_4': 'Ciència', 'solution_words_len': [10], 'solution_pattern': "__________", } ``` ### Prompt template The `hints` and `keywords` from the example above are injected in prompts such as: ``` Let’s play a game in Catalan! Your goal is to find a "solution" word or words of a specific length, which will be given to you. You will also receive four numbered hints. Try to solve each one to get a "hint word", which doesn’t need to match the solution’s length. Then, try to think about the common theme or connection between the "hint words" that you found. The final solution should fit the required letter count and is related to the "hint words" you identified. It is important to note that you don’t necessary need all the "hint words" to get to the final solution. If you’re struggling with a hint or suspect your guess might be wrong, it’s often better to focus on the "hint words" you are sure about. A wrong one can send you down the wrong path! Here are your hints: 1. {hint_1} 2. {hint_2} 3. {hint_3} 4. {hint_4} The "solution" should fit here: {solution_pattern}. What’s your guess? Return a JSON object with the following fields: ’hint_word_1’, ’hint_word_2’, ’hint_word_3’, ’hint_word_4’, ’solution’. If you could not find the word associated to some hint, simply keep that field as ’unknown’. ``` Refer to Appendix A of the [paper](https://aclanthology.org/2026.mme-main.4/) for a comprehensive overview of the prompt formats employed. ## Dataset Creation ### Curation Rationale The dataset was curated to evaluate multi-step reasoning and deeply rooted cultural knowledge; capabilities that are rarely, if ever, assessed by existing benchmarks. Each instance is carefully designed as a game-based riddle, requiring models to connect seemingly unrelated clues through reasoning and cultural understanding. The source data producers held collaborative sessions to develop the puzzles, always aiming to: - ensure topical diversity, avoiding repetition across instances, - vary difficulty levels by including at least one relatively easy clue to support solvability, - maintain linguistic and semantic coherence while encouraging creative associations. Then, our human annotators manually reviewed the original dataset to ensure that all instances can be interpreted in isolation, enhancing the fairness and reproducibility of the task. During this curation process, we also revised instances that were time-dependent or game-specific, as these relied on information unavailable to both LLMs and human annotators. Please refer to the [paper](https://aclanthology.org/2026.mme-main.4/) for further details. ### Who are the source data producers? The dataset was created by the original authors of the Vinclat game, three native Catalan speakers who released a new instance per day on the [official website](https://vinclat.cat/) (no longer available). ### Who are the annotators? Members of BSC's Annotation Team. All annotators were native Catalan speakers, born and raised in Catalonia, and held university degrees, ensuring both linguistic proficiency and cultural familiarity with the game and its underlying associations. Within the team, some annotators curated and crafted the dataset instances, while others established a human baseline and carried out a qualitative analysis of model responses. ### Personal and Sensitive Information No personal or sensitive information is included in the dataset. ### Bias, Risks, and Limitations The dataset is not expected to introduce social, cultural, or demographic biases and poses no known risks. ## Additional information ### Funding This work has been promoted and financed by: - Generalitat de Catalunya through the [Aina project](https://projecteaina.cat/). - Ministerio para la Transformación Digital y de la Función Pública - funded by the EU through Next GenerationEU – within the framework of the project [Desarrollo de Modelos ALIA](https://alia.gob.es/eng). ### License This work is licensed under the [Creative Commons Attribution 4.0 International License (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). ### Citation ```bibtext @inproceedings{pamies2026vinclat, title={Vinclat: Evaluating Reasoning, Cognition and Culture in One Game}, author={Pamies, Marc and Aula-Blasco, Javier and Gonzalez-Agirre, Aitor and Villegas, Marta}, booktitle={Proceedings of the First Workshop on Multilingual Multicultural Evaluation}, pages={49--66}, year={2026} } ``` ### Dataset Card Author - Marc Pàmies Massip ### Dataset Card Contact <[e-mail]>--- pretty_name: Vinclat 语言：加泰罗尼亚语 许可协议：CC-BY-4.0 任务类别： - 文本生成 - 问答 样本规模： - 样本量少于1000 标注创作者： - 专家生成 语言创作者： - 专家生成 标签： - 基准测试 - 评估 - 加泰罗尼亚语 - 加泰罗尼亚地区 - 游戏 - 谜语 数据集信息： 数据集总大小：1000 数据划分： - 划分名称：train 样本数量：1000 数据集特征： - 特征名：id 数据类型：int32 - 特征名：hint_1 数据类型：字符串 - 特征名：hint_2 数据类型：字符串 - 特征名：hint_3 数据类型：字符串 - 特征名：hint_4 数据类型：字符串 - 特征名：keyword_1 数据类型：字符串 - 特征名：keyword_2 数据类型：字符串 - 特征名：keyword_3 数据类型：字符串 - 特征名：keyword_4 数据类型：字符串 - 特征名：solution_words_len 数据类型：int64类型整数序列 - 特征名：solution_pattern 数据类型：字符串 --- # Vinclat 数据集卡片 **Vinclat** 是一款面向加泰罗尼亚语的多步推理数据集，采用游戏化结构评估大语言模型（Large Language Model, LLM）的推理能力与文化知识储备。 - **语言支持**：加泰罗尼亚语 - **相关论文**：[ACL文集](https://aclanthology.org/2026.mme-main.4/) - **排行榜**：[Hugging Face Space](https://huggingface.co/spaces/projecte-aina/vinclat-leaderboard) - **代码仓库**：[GitHub](https://github.com/mapama247/vinclat_eval) - **许可协议**：[CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/deed.en) - **资助方**：[Projecte Aina项目](https://projecteaina.cat/en/) - **数据集整理方**：巴塞罗那超级计算中心（Barcelona Supercomputing Center, BSC） - **共享方**：巴塞罗那超级计算中心（Barcelona Supercomputing Center, BSC） ## 数据集概述 **Vinclat** 是一款加泰罗尼亚语评估数据集，旨在评测大语言模型的多步推理能力与根植于文化的知识储备。该数据集包含1000条由人类标注专家精心打造并审核通过的样本。每条样本均采用游戏化谜语结构，要求模型通过多步推理过程破解谜题。给定四条独立线索后，模型需推断出看似毫无关联的中间概念，并通过创造性关联推导得出最终解决方案。成功完成该任务需要融合多种推理策略与语言理解能力，将逻辑推理与加泰罗尼亚语言及本土文化知识相结合。 为保障该基准测试的长期有效性，本次发布的数据集未提供最终解决方案，而是由作者维护外部排行榜，持续更新前沿模型的评估结果。该方案可有效避免基准测试污染，确保数据集始终作为可靠的评估资源。欢迎希望将自身模型纳入排行榜的研究人员联系作者。 ## 数据集结构 ### 数据字段说明 - `id`（int32）：每个样本的唯一标识符 - `hint_1`（str）：第一条解谜线索 - `hint_2`（str）：第二条解谜线索 - `hint_3`（str）：第三条解谜线索 - `hint_4`（str）：第四条解谜线索 - `keyword_1`（str）：与第一条线索关联的概念 - `keyword_2`（str）：与第二条线索关联的概念 - `keyword_3`（str）：与第三条线索关联的概念 - `keyword_4`（str）：与第四条线索关联的概念 - `solution_words_len`（list[int64]）：按顺序表示解决方案各单词字母数的整数列表 - `solution_pattern`（str）：解决方案的可视化格式，用下划线代表每个字母，单词间以空格分隔 ### 样本示例 { 'id': 1, 'hint_1': 'El sufix de la majoria d’adverbis', 'hint_2': 'La platja més surfista de Cadis', 'hint_3': 'De Shock o de parella', 'hint_4': 'La filla de la paciència', 'keyword_1': 'Ment', 'keyword_2': 'Tarifa', 'keyword_3': 'Teràpia', 'keyword_4': 'Ciència', 'solution_words_len': [10], 'solution_pattern': "__________", } ### 提示词模板 上述示例中的`hints`与`keywords`会被注入至如下提示模板中： 让我们一起来玩一局加泰罗尼亚语的游戏！你的目标是找到符合特定长度要求的一个或多个"解决方案"单词，该长度要求会提前提供给你。同时你还会收到四条编号的线索，请逐一破解以获得对应的"线索单词"，该单词的长度无需与解决方案一致。随后，请思考你找到的这些"线索单词"之间的共同主题或关联。最终的解决方案需要符合指定的字母数要求，且与你识别出的"线索单词"相关。 需要注意的是，你未必需要用到所有的"线索单词"来推导出最终解决方案。如果你在某条线索上卡住，或者怀疑自己的猜测有误，通常优先聚焦于你确信正确的"线索单词"会更有效。错误的线索可能会将你引向歧途！ 以下是你的线索： 1. {hint_1} 2. {hint_2} 3. {hint_3} 4. {hint_4} "解决方案"应符合如下格式：{solution_pattern}。请给出你的猜测？请返回一个JSON对象，包含以下字段：'hint_word_1'、'hint_word_2'、'hint_word_3'、'hint_word_4'、'solution'。如果你无法推导某条线索对应的单词，请将该字段设为'unknown'。 请参阅[论文](https://aclanthology.org/2026.mme-main.4/)的附录A以了解所用提示格式的完整说明。 ## 数据集创建 ### 整理原则 本数据集旨在评估多步推理与根植于本土文化的知识储备，这类能力在现有基准测试中极少被覆盖。每条样本均被精心设计为游戏化谜语，要求模型通过推理与文化理解关联看似无关的线索。原始数据创作者通过协作会话开发谜题，始终遵循以下原则： 1. 确保主题多样性，避免样本间重复； 2. 通过至少设置一条相对简单的线索调整难度，保障任务可解性； 3. 维持语言与语义连贯性，同时鼓励创造性联想。 随后，人类标注人员对原始数据集进行人工审核，确保所有样本均可独立解读，提升任务的公平性与可复现性。在审核过程中，我们还修订了依赖时效性信息或特定游戏背景的样本，这类信息对大语言模型与人类标注者均不可得。详细信息请参阅[论文](https://aclanthology.org/2026.mme-main.4/)。 ### 原始数据创作者 本数据集由Vinclat游戏的原作者创建，共三名母语为加泰罗尼亚语的创作者，他们曾在[官方网站](https://vinclat.cat/)（现已停用）每日发布一则新谜题。 ### 标注人员 标注团队成员均来自巴塞罗那超级计算中心的标注团队。所有标注者均为在加泰罗尼亚地区出生长大的母语使用者，且拥有大学学历，确保具备足够的语言能力与对游戏及相关关联的文化熟悉度。团队中部分成员负责制作与整理数据集样本，其余成员则建立了人类性能基线，并对模型响应进行定性分析。 ### 个人与敏感信息 本数据集未包含任何个人或敏感信息。 ### 偏见、风险与局限性 本数据集预计不会引入社会、文化或人口偏见，且无已知风险。 ## 附加信息 ### 资助情况 本工作由以下机构资助： - 加泰罗尼亚政府通过[Aina项目](https://projecteaina.cat/)； - 西班牙数字化转型与公共职能部通过欧盟下一代EU计划，在[ALIA模型开发项目](https://alia.gob.es/eng)框架下资助。 ### 许可协议 本作品采用[知识共享署名4.0国际许可协议（CC BY 4.0）](https://creativecommons.org/licenses/by/4.0/)进行许可。 ### 引用格式 bibtex @inproceedings{pamies2026vinclat, title={Vinclat: Evaluating Reasoning, Cognition and Culture in One Game}, author={Pamies, Marc and Aula-Blasco, Javier and Gonzalez-Agirre, Aitor and Villegas, Marta}, booktitle={Proceedings of the First Workshop on Multilingual Multicultural Evaluation}, pages={49--66}, year={2026} } ### 数据集卡片作者 - Marc Pàmies Massip ### 数据集卡片联系方式 <[e-mail]>