Evaluation of Vanilla and RAG-Supported LLM Comprehension of the EU AI Act

This repository contains the benchmark dataset, retrieval corpus, evaluation results, and code accompanying the article: Tsalera, E.; Pagiatakis, G.; Papadakis, A. Evaluation of Vanilla and RAG-Supported LLM Comprehension of the EU AI Act. Future Internet 2026, 18(9), 469. https://doi.org/10.3390/fi18090469 Overview The study evaluates the regulatory comprehension of four open-source large language models — TinyLlama-1.1B-Chat, Gemma-2B-Instruct, Mistral-7B-Instruct-v0.3, and Llama-3.1-8B-Instruct — on the EU Artificial Intelligence Act (Regulation (EU) 2024/1689). Each model is tested under two conditions: vanilla inference (parametric knowledge only) and Retrieval-Augmented Generation (RAG), in which relevant passages of the Act are retrieved via a FAISS-indexed all-MiniLM-L6-v2 sentence-embedding pipeline with standardized top-3 chunk retrieval applied uniformly across all models. Contents data/ — the benchmark of 100 multiple-choice questions in JSON, JSONL, and CSV. Each question includes the stem, four options (A–D), the correct answer, three classification labels (cognitive difficulty, knowledge type, regulatory mechanism), and the Article/Recital/Annex source reference. results/ — per-question model outputs under both conditions and a performance-metrics table (accuracy per model/condition and McNemar paired-test results). knowledge_base/ — the chunked retrieval corpus (786 chunks, 180 words, 20-word overlap) and a note on the source text of the Act. code/ — evaluation and statistics scripts, the inference/RAG pipeline configuration, and the prompt templates. The benchmark is classified across three orthogonal dimensions. Statistical significance of the RAG effect is assessed with McNemar's paired test, and retrieval quality is reported via Precision@3, Recall@3, and MRR against an expert relevance judgement set. Citation Please cite both this Zenodo record and the associated article (Future Internet 2026, 18(9), 469; https://doi.org/10.3390/fi18090469).