Prompts generated from ChatGPT3.5, ChatGPT4, LLama3-8B, and Mistral-7B with NYT and HC3 topics in different roles and parameters configurations

Description

Prompts generated from ChatGPT3.5, ChatGPT4, Llama3-8B, and Mistral-7B with NYT and HC3 topics in different roles and parameter configurations. 

The dataset is useful to study lexical aspects of LLMs with different parameters/roles configurations.

The 0_Base_Topics.xlsx file lists the topics used for the dataset generation

The rest of the files collect the answers of ChatGPT to these topics with different configurations of parameters/context:

Temperature (parameter): Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.

Frequency penalty (parameter): Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.

Top probability (parameter): An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass.

Presence penalty (parameter): Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.

Roles (context)

Default: No role is assigned to the LLM, the default role is used.

Child: The LLM is requested to answer as a five-year-old child. 

Young adult male: The LLM is requested to answer as a young male adult.    

Young adult female: The LLM is requested to answer as a young female adult.    

Elderly adult male: The LLM is requested to answer as an elderly male adult.    

Elderly adult female: The LLM is requested to answer as an elderly female adult.

Affluent adult male: The LLM is requested to answer as an affluent male adult.    

Affluent adult female: The LLM is requested to answer as an affluent female adult. 

Lower-class adult male: The LLM is requested to answer as a lower-class male adult. 

Lower-class adult female: The LLM is requested to answer as a lower-class female adult.  

Erudite: The LLM is requested to answer as an erudite who uses a rich vocabulary.

Paper

Paper: Beware of Words: Evaluating the Lexical Diversity of Conversational LLMs using ChatGPT as Case Study

Cite:

@article{10.1145/3696459,author = {Mart\'{\i}nez, Gonzalo and Hern\'{a}ndez, Jos\'{e} Alberto and Conde, Javier and Reviriego, Pedro and Merino-G\'{o}mez, Elena},title = {Beware of Words: Evaluating the Lexical Diversity of Conversational LLMs using ChatGPT as Case Study},year = {2024},publisher = {Association for Computing Machinery},address = {New York, NY, USA},issn = {2157-6904},url = {https://doi.org/10.1145/3696459},doi = {10.1145/3696459},abstract = ,note = {Just Accepted},journal = {ACM Trans. Intell. Syst. Technol.},month = sep,keywords = {LLM, Lexical diversity, ChatGPT, Evaluation}}