Undergraduate Pacific Studies Exam Generation and Answering Using Retrieval Augmented Generation and Large Language Models

This repository contains the source code and data used for evaluating RAG versus non-RAG performance of OpenAI models. It includes: A Jupyter notebook (RAG_Assistant_and_Metrics.ipynb) implementing the RAG (Retrieval-Augmented Generation) assistants and evaluating performance using various metrics. The notebook uses two models (GPT-3.5-Turbo and GPT-4-Turbo) with access to an undergraduate textbook to generate true/false, multiple choice, and long-form questions. Four models are then evaluated: two with textbook access (RAG-enabled GPT-3.5-Turbo and GPT-4-Turbo), and two without (ChatGPT-3.5-Turbo and ChatGPT-4-Turbo). Their responses are scored against the answer key using a variety of metrics to quantify the benefit of RAG for factual accuracy, semantic similarity, and overall response quality: ROUGE-1 from the rouge_score package cosine_similarity from sklearn.metrics.pairwise Embedding similarity using spaCy's transformer-based embeddings Merged results (EXAMS.csv) combining 14 separate exam datasets with answer keys, retrieved textbook excerpts, and responses from each model. A final metrics summary (score_results.csv) representing performance scores.