Speed and Conversational Large Language Models (LLMs): Not All Is About Tokens per Second

Speed and Conversational Large Language Models (LLMs): Not All Is About Tokens per Second This repository contains the results of benchmarking different language models across three main NLP tasks including the time of response, the raw response and the number of tokens: Task 1: Answering test-style questions. Task 2: Paraphrasing (text reformulation). Task 3: Multiple choice question answering. Each file has the following structure: task{N}_benchmark_{MODEL}_{BITS}. where: {N} → task number (1, 2, or 3). {MODEL} → the model tested (e.g., Llama-2-7b-chat-hf). {BITS} → model precision (e.g., 16bits). File Structure Examples of available combinations in this repository: Task 1 task1_benchmark_Llama-2-7b-chat-hf_16bits.xlsx task1_benchmark_Meta-Llama-3-8B-Instruct_16bits.xlsx task1_benchmark_Mistral-7B-Instruct-v0.1_16bits.xlsx task1_benchmark_Yi-6B-Chat_16bits.xlsx task1_benchmark_gemma-7b-it_16bits.xlsx Task 2 task2_benchmark_Llama-2-7b-chat-hf_16bits.xlsx task2_benchmark_Meta-Llama-3-8B-Instruct_16bits.xlsx task2_benchmark_Yi-6B-Chat_16bits.xlsx task2_benchmark_gemma-7b-it_16bits.xlsx Task 3 task3_benchmark_Llama-2-7b-chat-hf_16bits.xlsx task3_benchmark_Meta-Llama-3-8B-Instruct_16bits.xlsx task3_benchmark_Mistral-7B-Instruct-v0.1_16bits.xlsx task3_benchmark_Yi-6B-Chat_16bits.xlsx task3_benchmark_gemma-7b-it_16bits.xlsx File Template Each file follows a common structure with at least the following fields: Input → Prompt given to the model Model → Model name Raw Output → Unprocessed raw response Tokens Used → Total tokens consumed/generated Execution Time (s) → Total inference duration Models Evaluated The following language models were benchmarked: Llama-2-7b-chat-hf Meta-Llama-3-8B-Instruct Mistral-7B-Instruct-v0.1 Yi-6B-Chat gemma-7b-it All were executed in 16-bit precision. Reference If you use this repository, please cite the following article: Conde, J., González, M., Reviriego, P., Gao, Z., Liu, S., & Lombardi, F. (2024).Speed and Conversational Large Language Models: Not All Is About Tokens per Second.Computer, 57(8), 74-80.doi: 10.1109/MC.2024.3399384 BibTeX entry: @ARTICLE{ 10632720, author={Conde, Javier and González, Miguel and Reviriego, Pedro and Gao, Zhen and Liu, Shanshan and Lombardi, Fabrizio}, journal={Computer}, title={Speed and Conversational Large Language Models: Not All Is About Tokens per Second}, year={2024}, volume={57}, number={8}, pages={74-80}, keywords={Large language models;Chatbots;Artificial intelligence;User interfaces;Business;Open source software;Source coding;Performance evaluation;Costs;Investment;Modeling}, doi={10.1109/MC.2024.3399384} }