Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratings

Psycholinguistics framework It includes the results of the experiment and the framework with the code. English Familiarity results: It is located in the folder familiarity_english Framework code: Features Run esitamtion of word features and fine-tunings using OpenAI models, Gemini models, Hugging Face models using HF API or local deployment Fine-tune new versions of LLMs Setting Up the framework Copy apis_example.env to apis.env and fill it with your model credentials Install dependencies with pip install -r requirements.txt Setting Up a New Experiment Initial configuration Create a folder with the name of the experiment, add the data in XLSX format in the data subfolder, and possible prompts in TXT format in the prompts subfolder. Create a config.yaml file in the experiment folder using the content of config_example.yaml as a template. Make estimations with some model (including fine-tuning models): Prepare the experiment by running python3 prepare_experiment.py <EXPERIMENT_PATH> <EXPERIMENT NAME>. -> generates the batches Save the batches.jsonl files in batches folder Run the experiment by executing python3 execute_experiment.py <EXPERIMENT_PATH> <EXPERIMENT NAME>. -> executes the batches. Processing results: Save the results of batches in results folder Combine all the batches in a file called batches.jsonl Combine all the results files in a file called results.jsonl You can use these commands: cat batches/*.jsonl >> batches/batches.jsonl cat results/*.jsonl >> results/results.jsonl Execute python3 generateResults.py <EXPERIMENT_PATH> {json, num, weighted_sum} [extra-otpions] for the experiment -> for your own experiment is possible that you want to modify it. We recommend creating a new file for reproducibility -> it generates a .xlsx with the results Make a fine-tuning: Prepare the fine-tuning dataset by running python3 create_finetuning_dataset.py <EXPERIMENT_PATH> <EXPERIMENT NAME>. Fine-tune the model by running python3 execute_finetune.py <EXPERIMENT_PATH> <FT_NAME>. Calculate the correlation by running python3 calculate_correlation.py <EXPERIMENT_PATH>. To test the finetuning execute the same steps mentioned before. Other scripts: execute_individual_api_{google/openai}.py: Instead of batch operations executes individual calls to the model API combine_excels.py: It makes a left join of file1.xlsx and file2.xlsx using two columns and generates a new excel with the same name as file1.xlsx (saving the initial one as file1_old.xlsx) python3 combine_excels.py <file1.xlsx> <file2.xlsx> <Column-file1,Column-file2> <suffix for columns of file2.xlsx>