AuReal: an Automated Model for LLM Reliability Evaluation and Hallucination Detection

This dataset was constructed to support the evaluation of Large Language Model (LLM) reliability, focusing on correctness and consistency across repeated queries, including five real-world datasets, i.e., ARC, CommonsenseQA (CQA), BoolQ (BQA), SQuAD (SQA) and NarrativeQA (NQA). It contains question\u2013answer pairs where each question is queried multiple times (2\u20135) against different LLMs. Each response is annotated with a binary correctness label by comparing it with ground-truth answers, and an aggregated reliability label indicates whether all responses to a question are correct. To facilitate consistency analysis, semantic similarity scores are also provided, computed using BERT-based embeddings between answers and their corresponding questions. These annotations enable fine-grained evaluation of LLM reliability, particularly in scenarios involving correctness, viewpoint-level consistency, and hallucination detection. The dataset is released in CSV format and can serve as a benchmark for developing and testing automated methods for reliability and safety assessment of LLMs.