Automotive Cybersecurity QA and RAG Evaluation Dataset

Automotive Cybersecurity QA and RAG Evaluation Dataset

This dataset provides a collection of human-curated question–answer pairs in the automotive cybersecurity domain, together with model-generated answers obtained using multiple retrieval-augmented generation (RAG) configurations. It is designed to support research on domain-specific question answering, retrieval-augmented generation, and the evaluation of large language models in automotive cybersecurity. Additional details on the RAG configurations, retrieval strategies, and evaluation methodology are described in the associated publication.

The dataset includes:

A set of human-written question–answer pairs created by domain experts, serving as ground truth.

Model-generated answers produced under four different RAG configurations: ensemble, semantic, syntactic, and two-stage.

Dataset Structure

human_answers/: Contains human-written reference answers.

answers_rag/: Contains model-generated answers, organized by RAG configuration (ensemble, semantic, syntactic, two_stage). Each folder contains CSV files corresponding to different evaluated language models.

Data Format

All files follow the same schema:

q_id: Unique identifier of the question

question: The natural language question

answer: The corresponding answer (human or model-generated depending on the file)

Purpose

This dataset can be used to:

Benchmark large language models on domain-specific question answering

Evaluate and compare different RAG strategies

Analyze and compare human-written and model-generated answers

Support research on LLM applications in automotive cybersecurity