Vocabulary Tests LLMs

Vocabulary evaluation of LLMs

This repository contains the results for the different vocabulary tests run on LLM tools/models presented in the paper: "The continued usefulness of vocabulary tests for evaluating large language models" currently published  in PLOS ONE: https://doi.org/10.1371/journal.pone.0308259

The name of the files correspond to the vocabulary tests for which results are presented in Tables 3-6 in the paper (note that questions for the TOEFL test are not public).  

In each file, the first column has the question posed to the LLM tool/model, followed by the correct answer. The rest of the columns correspond to the answers of the different LLM tools/models evaluated.  The results and percentages are summarized at the bottom of the file after the last test items.

The models evaluated are:

Model
Link

Llama 2 7b  
https://huggingface.co/meta-llama/Llama-2-7b-chat

Llama 2 13b  
https://huggingface.co/meta-llama/Llama-2-13b-chat

Llama 2 70b
https://huggingface.co/meta-llama/Llama-2-70b-chat

Mistral 7b v0.1
https://huggingface.co/mistralai/Mistral-7B-v0.1

GPT 3.5 turbo 0613
https://platform.openai.com/docs/models/gpt-3-5-turbo

GPT 4 0613  
https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4

Bard

To cite our work:

@article{10.1371/journal.pone.0308259,
doi = {10.1371/journal.pone.0308259},
author = {Martínez, Gonzalo AND Conde, Javier AND Merino-Gómez, Elena AND Bermúdez-Margaretto, Beatriz AND Hernández, José Alberto AND Reviriego, Pedro AND Brysbaert, Marc},
journal = {PLOS ONE},
publisher = {Public Library of Science},
title = {Establishing vocabulary tests as a benchmark for evaluating large language models},
year = {2024},
month = {12},
volume = {19},
url = {https://doi.org/10.1371/journal.pone.0308259},
pages = {1-17},
number = {12},

}