Dataset for "LLM-aided explanations of EDA synthesis errors"

Dataset for "LLM-aided explanations of EDA synthesis errors"

Authors: Siyu Qiu, Benjamin Tan, Hammond Pearce

This Zenodo contains the open-source data used for the ISLAD submission "LLM-aided explanations of EDA synthesis errors" which aimed to use OpenAI LLMs for generating novice-focused explanations of common synthesis errors.

Error explanations for RTL and HDL Code with OpenAI's LLMs

Welcome to our error explanation tool for RTL and HDL code in Verilog and VHDL! This repository helps you generate error explanations for your code using OpenAI's models, making it easier to find and fix bugs.

Directory Structure

- `new_structure/`: Contains all bugs in separate files, along with labelling CSV files that match the LLM responses for each bug.

- `bug_id/`: Each bug has responses from gpt-3.5-turbo (40 responses), gpt-4 (4 responses), and gpt-4-turbo-preview (4 responses).

- `rtl/`: Includes RTL code used for bugs, compatible with both Quartus and Vivado.

- `Quartus/`: Contains Quartus project files (.qpf) and constraints files (.qsf).

- `Vivado/`: Holds Vivado project files (.xpr) and constraints files (.xdc).

- `llm_responses/`: Contains CSV files with records of the generated LLM responses for each bug.

- `labelling/`: Contains CSV files with manually scored metrics for evaluating the LLM responses.

- `error_list.csv`: This file lists all bugs with details like bug id, type, IDE, file name, language, and error message. main.py uses this file to process bugs.

- `main.py`: Use this script to generate error explanations for your RTL and HDL code with OpenAI's models. It reads bug information from error_list.csv, loads the buggy code, and interacts with the OpenAI API to get explanations for the bugs.

- `try.py`: This script defines dictionaries to store statistics about bug evaluations. It loads bug data from CSV files, processes each file to update the statistics, and prints a summary table using the tabulate library.

The labelling directories contains CSV files with metrics for evaluating the LLM responses. These metrics include Conceptual Accuracy, Inaccuracy, Relevance, Completeness, and Solution Provided. try.py processes these files to generate summary statistics.

Important note:
You need to create an API key for OpenAI and save it as a file called OPENAI_TOKEN in the root project directory. gitignore will ignore this file.