LLMWorldOfWords/LWOW: First release

The "LLM World of Words" (LWOW) [1] is a collection of datasets of English free association norms generated by various large language models (LLMs). Currently, the collection consists of datasets generated by Mistral, LLaMA3, and Claude Haiku. The datasets are modeled after the "Small World of Words" (SWOW) (https://smallworldofwords.org/en/project/) [2] English free association norms, generated by humans, consisting of over 12,000 cue words and over 3 million responses. The purpose of the LWOW datasets is to provide a way to investigate various aspects of the semantic memory of LLMs using an approach that has been applied extensively for investigating the semantic memory of humans. These datasets, together with the SWOW dataset, can be used to gain insights about similarities and differences in the language structures possessed by humans and LLMs. What are free associations? Free associations are implicit mental connections between words or concepts. They are typically accessed by presenting humans (or AI agents) with a cue word and then asking them to respond with the first words that come to mind. The responses represent implicit associations that connect different concepts in the mind, reflecting the semantic representations that underly patterns of thought, memory, and language. For example, given the cue word "woman", a common free association response might be "man", reflecting the associative mental relation between these two concepts. How can they be used? Free associations have been extensively used in cognitive psychology and linguistics as a tool for studying language and cognitive information processing. They provide a way for researchers to understand how conceptual knowledge is organized and accessed in the mind. Free associations are often used to built network models of semantic memory by connecting cue words to their responses. When thousands of cues and responses are connected in this way, the result is a complex network model that represents the complex organization of semantic knowledge. Such models enable the investigation of complex cognitive processes that take place within semantic memory, and can be used to study a variety of cognitive phenomena such as language learning, creativity, personality traits, and cognitive biases. Validation of the datasets with semantic priming The LWOW datasets were validated using data from the Semantic Priming Project (https://www.montana.edu/attmemlab/spp.html) [3], which implements a lexical decision task (LDT) to study semantic priming. The semantic priming effect is the cognitive phenomenon that a target word (e.g. nurse) is more easily recognized when it is prompted by a related prime word (e.g. doctor) compared to an unrelated prime word (e.g. doctrine). We simulated the semantic priming effect within network models of semantic memory built from both the LWOW and the SWOW free association norms by implementing spreading activation processes within the networks [4]. We found that the final activation levels of prime-target pairs correlated significantly with reaction time data for the same prime-target pairs from the LDT. Specifically, the activation of a target node (e.g. nurse) is higher when a related prime node (e.g. doctor) is activated compared to an unrelated prime node (e.g. doctrine). These results demonstrate how the LWOW datasets can be used for investigating cognitive and linguistic phenomena in LLMs, demonstrating the validity of the datasets. Investigating gender biases To demonstrate how this dataset can be used to investigate gender biases in LLMs compared to humans, we conducted an analysis using network models of semantic memory built from both the LWOW and the SWOW free association norms. We applied a methodology that simulates semantic priming within the networks to measure the strength of association between pairs of concepts, for example, "woman" and "forecful" vs. "man" and "forceful". We applied this methodology using a set of female-related and male-related primes, and a set of female-related and male-related targets. This analysis revealed that certain adjectives like "forceful" and "strong" are more strongly associated with certain genders, shedding light on the types of stereotypical gender biases that both humans and LLMs possess. Technical notes The free associations were generated (either via API or locally, depending on the LLM) by providing each LLM with a set of cue words and the following prompt: "You will be provided with an input word. Write the first 3 words you associate to it separated by a comma." This prompt was repeated 100 times for each cue word, resulting in a dataset of 11,545 unique cues words and 3,463,500 total responses for each LLM. How to access and use the datasets The LWOW datasets for Mistral, Llama3, and Haiku can be found in the LWOW_datasets folder, which contains two subfolders. The .csv files of the processed cues and responses can be found in the processed_datasets folder while the .csv files of the edge lists of the semantic networks constructed from the datasets can be found in the graphs/edge_lists folder. Since the LWOW datasets are intended to be used in comparison to humans, we have further processed the original SWOW dataset to create a Human dataset that is aligned with the processing that we applied to the LWOW datasets. While this human dataset is not included in this repository due to the license of the original SWOW dataset, it can be easily reproduced by running the code provided in the reproducibility folder. We highly encourage you to generate this dataset as it enabales a direct comparison between humans and LLMs. The Human dataset can be generated with the following steps: Go to the SWOW research page (https://smallworldofwords.org/en/project/research) [2] and download the English processed data (SWOW-EN18). Save this .csv file with the name "SWOW-EN.R100.csv" in the reproducibility/data/original_datasets folder. Run the python file FA_data_Cleaning.py saved in the reproducibility folder. This will generate a .csv of the processed Human dataset, which will be saved in the reproducibility/data/processed_datasets folder. Note that this python script will also regenerate the .csv files of the processed LWOW datasets (the same that can be found in the LWOW_datasets/processed_datasets folder). Run the python file FA_build_Networks.py saved in the reproducibility folder. This will generate a .csv of the edge list of the semantic network constructed from the Human dataset, which will be saved in the reproducibility/data/graphs/edge_lists folder. Note that this python script will also regenerate the .csv files of the same edges lists of the LLM networks (the same that can be found in the LWOW_datasets/graphs/edge_lists folder). This python script will also produce igraph versions of all the semantic networks. How to reproduce the data and analyses To reproduce the analyses, first the required external files need to be downloaded: Go to the SWOW research page (https://smallworldofwords.org/en/project/research) [2] and download the English data SWOW-EN18. Save this .csv file with the name "SWOW-EN.R100.csv" in the reproducibility/data/original_datasets folder. Go to the Semantic Priming Project (https://www.montana.edu/attmemlab/spp.html) [3] and download the LDT Priming Data. Save this .csv file with the name "primingLDT_data.csv" in the reproducibility/data/LDT_analyses folder. Once the files are saved in the correct folders, follow the instructions in each script, which can be found in the reproducibility folder. The scripts should be run in the following order: FA_data_Generation.py: generates the raw LLM datasets FA_data_Cleaning.py: processes the original SWOW dataset and the raw LLM datasets FA_build_Networks.py: builds the semantic networks from the datasets FA_analyses_LDT_Gender.py and FA_spreadr.r: implements spreading activation processes within the networks in order to validate the datasets and investigate gender biases Do you want to know more? Read the Preprint! Abramski, K., et al. (2024). The "LLM World of Words" English free association norms generated by large language models (https://arxiv.org/abs/2412.01330) Funding & Legal SoBigData.it which receives funding from the European Union – NextGenerationEU – National Recovery and Resilience Plan (Piano Nazionale di Ripresa e Resilienza, PNRR) – Project: “SoBigData.it – Strengthening the Italian RI for Social Mining and Big Data Analytics” – Prot. IR0000013 – Avviso n. 3264 del 28/12/2021; EU NextGenerationEU programme under the funding schemes PNRR-PE-AI FAIR (Future Artificial Intelligence Research). The HumaneAI-Net project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 952026. COGNOSCO grant funded by Università di Trento (Grant ID: PS 22_27). For speaking requests and enquiries, please contact: Katherine Abramski : [e-mail] Giulio Rossetti : [e-mail] Massimo Stella : [e-mail] References [1] Abramski, K., et al. (2024). The" LLM World of Words" English free association norms generated by large language models. arXiv preprint arXiv:2412.01330. [2] De Deyne, S., et al. (2019). The “Small World of Words” English word association norms for over 12,000 cue words. Behavior research methods, 51, 987-1006. [3] Hutchison, K. A. et al. (2013). The semantic priming project. Behavior research methods, 45, 1099-1114. [4] Siew, C. S. (2019). spreadr: An R package to simulate spreading activation in a network. Behavior Research Methods, 51(2), 910-929.“LLM词语世界”（LLM World of Words, LWOW）[1] 是由各类大语言模型（Large Language Model, LLM）生成的英语自由联想规范数据集集合。目前该集合包含由Mistral、LLaMA3与Claude Haiku生成的数据集。这些数据集以人类生成的“小世界词语”（Small World of Words, SWOW）英语自由联想规范为蓝本，该蓝本包含超过12000个提示词与超300万条回复，其官方网址为https://smallworldofwords.org/en/project/ [2]。LWOW数据集的设计初衷是，借助已广泛用于人类语义记忆研究的范式，为探究大语言模型语义记忆的多个维度提供支持。该数据集与SWOW数据集相结合，可用于洞察人类与大语言模型所拥有的语言结构的异同。 何为自由联想？ 自由联想是词语或概念之间的内隐心理联结。通常的实验范式为向人类（或AI智能体（AI Agent））呈现一个提示词，要求其以脑海中浮现的首个词语进行回复。这些回复代表了联结不同概念的内隐联想，反映了支撑思维、记忆与语言模式的语义表征。例如，当提示词为“woman”时，常见的自由联想回复可能是“man”，体现了这两个概念之间的联想心理关系。 自由联想有何应用价值？ 自由联想已在认知心理学与语言学中被广泛用作研究语言与认知信息处理的工具。它为研究者提供了一条理解心智中概念知识的组织与提取方式的路径。自由联想常被用于构建语义记忆的网络模型，即将提示词与其回复进行联结。当数千个提示词与回复以此方式联结后，最终会形成一个复杂的网络模型，用以表征语义知识的复杂组织架构。此类模型可用于探究语义记忆内部发生的复杂认知过程，也可用于研究各类认知现象，如语言学习、创造力、人格特质与认知偏差等。 基于语义启动的数据集验证 LWOW数据集通过语义启动项目（Semantic Priming Project）的数据完成了有效性验证[3]，该项目通过词汇判断任务（lexical decision task, LDT）研究语义启动效应。语义启动效应是一种认知现象：当目标词（如“nurse”）由相关启动词（如“doctor”）提示时，其识别难度相较于无关启动词（如“doctrine”）更低。我们通过在由LWOW与SWOW自由联想规范构建的语义记忆网络中实现扩散激活过程[4]，模拟了语义启动效应。研究发现，启动词-目标词对的最终激活水平与词汇判断任务中相同启动词-目标词对的反应时数据显著相关。具体而言，当相关启动节点（如“doctor”）被激活时，目标节点（如“nurse”）的激活水平高于无关启动节点（如“doctrine”）的情况。上述结果证明了LWOW数据集可用于探究大语言模型的认知与语言现象，验证了该数据集的有效性。 探究性别偏见 为展示该数据集可用于对比探究人类与大语言模型中的性别偏见，我们分别基于LWOW与SWOW自由联想规范构建的语义记忆网络开展了分析。我们采用了一种在网络内模拟语义启动的方法，以衡量概念对之间的联结强度，例如“woman”与“forceful”相较于“man”与“forceful”的联结强度。我们使用一组与女性相关、一组与男性相关的提示词，以及一组与女性相关、一组与男性相关的目标词开展了该分析。该分析揭示，诸如“forceful”与“strong”等特定形容词与特定性别存在更强的联结，这有助于揭示人类与大语言模型所共有的刻板性别偏见类型。 技术说明 自由联想的生成方式为：向每个大语言模型提供一组提示词，并附带以下提示语：“你将收到一个输入词语，请写出你联想到的前3个词语，以逗号分隔。” 针对每个提示词，该提示语被重复执行100次，最终每个大语言模型将生成包含11545个唯一提示词与总计3463500条回复的数据集。 数据集的获取与使用方法 Mistral、Llama3与Haiku对应的LWOW数据集可在LWOW_datasets文件夹中找到，该文件夹包含两个子文件夹。经过处理的提示词与回复的.csv文件可在processed_datasets文件夹中获取，而由数据集构建的语义网络的边列表.csv文件则可在graphs/edge_lists文件夹中找到。 由于LWOW数据集旨在用于与人类数据进行对比，我们进一步对原始SWOW数据集进行了处理，生成了与LWOW数据集处理流程对齐的人类数据集。由于原始SWOW数据集的许可协议限制，该人类数据集未包含在本仓库中，但可通过运行reproducibility文件夹中提供的代码轻松复现。我们强烈建议生成该数据集，因其可实现人类与大语言模型的直接对比。人类数据集的生成步骤如下： 1. 访问SWOW研究页面（https://smallworldofwords.org/en/project/research）[2]，下载英语处理后数据（SWOW-EN18），将该.csv文件命名为“SWOW-EN.R100.csv”并保存至reproducibility/data/original_datasets文件夹。 2. 运行reproducibility文件夹中的FA_data_Cleaning.py文件，该脚本将生成经过处理的人类数据集的.csv文件，保存至reproducibility/data/processed_datasets文件夹。请注意，该Python脚本还将重新生成经过处理的LWOW数据集的.csv文件（与LWOW_datasets/processed_datasets文件夹中的文件一致）。 3. 运行reproducibility文件夹中的FA_build_Networks.py文件，该脚本将生成由人类数据集构建的语义网络的边列表.csv文件，保存至reproducibility/data/graphs/edge_lists文件夹。请注意，该Python脚本还将重新生成LLM网络的边列表.csv文件（与LWOW_datasets/graphs/edge_lists文件夹中的文件一致）。此外，该脚本还将生成所有语义网络的igraph版本。 数据与分析的复现方法 若要复现分析，首先需要下载所需的外部文件： 1. 访问SWOW研究页面（https://smallworldofwords.org/en/project/research）[2]，下载英语数据SWOW-EN18，将该.csv文件命名为“SWOW-EN.R100.csv”并保存至reproducibility/data/original_datasets文件夹。 2. 访问语义启动项目（https://www.montana.edu/attmemlab/spp.html）[3]，下载LDT启动数据，将该.csv文件命名为“primingLDT_data.csv”并保存至reproducibility/data/LDT_analyses文件夹。 待文件保存至正确文件夹后，按照每个脚本中的说明进行操作，所有脚本均可在reproducibility文件夹中找到。脚本的运行顺序如下： 1. FA_data_Generation.py：生成原始LLM数据集 2. FA_data_Cleaning.py：处理原始SWOW数据集与原始LLM数据集 3. FA_build_Networks.py：由数据集构建语义网络 4. FA_analyses_LDT_Gender.py与FA_spreadr.r：在网络中实现扩散激活过程，以验证数据集并探究性别偏见 想要了解更多？请阅读预印本！ Abramski, K. 等人（2024）。《由大语言模型生成的“LLM词语世界”英语自由联想规范》（https://arxiv.org/abs/2412.01330） 资助与法律声明 SoBigData.it 收到来自欧盟下一代欧盟（NextGenerationEU）——国家复苏与恢复计划（Piano Nazionale di Ripresa e Resilienza, PNRR）——的资助，项目名称：“SoBigData.it——强化意大利社会挖掘与大数据分析研究基础设施”，协议编号：IR0000013，公告编号：2021年12月28日第3264号； 欧盟下一代欧盟计划下的PNRR-PE-AI FAIR（未来人工智能研究）资助方案； HumaneAI-Net项目收到欧盟地平线2020研究与创新计划的资助，资助协议编号：952026； COGNOSCO项目由特伦托大学资助（资助编号：PS 22_27）。 如需报告与咨询，请联系： Katherine Abramski : [e-mail] Giulio Rossetti : [e-mail] Massimo Stella : [e-mail] 参考文献 [1] Abramski, K., 等人（2024）。《由大语言模型生成的“LLM词语世界”英语自由联想规范》。arXiv预印本 arXiv:2412.01330。 [2] De Deyne, S., 等人（2019）。《面向超过12000个提示词的“小世界词语”英语词语联想规范》。行为研究方法, 51, 987-1006。 [3] Hutchison, K. A. 等人（2013）。《语义启动项目》。行为研究方法, 45, 1099-1114。 [4] Siew, C. S.（2019）。spreadr：用于模拟网络中扩散激活的R包。行为研究方法, 51(2), 910-929。