Hydrangea

Hydrangea is a defect library for LLM-enabled software. Hydrangea has 4 main petals, each corresponding to one of the major components where defects often arise: LLM agent, vector database, software component, and system.This repository is made available under the Public Domain Dedication and License v1.0 whose full text can be found at: http://opendatacommons.org/licenses/pddl/1.0/ - see the LICENSE file for details.What is LLM-enabled software?It is software that integrates LLMs (large language models) with RAG (retrieval-augmented generation ) support to realize intelligence features.It contains four components:<b>LLM agent</b> that manages LLM interfaces, constructs prompts, and invokes the LLM<b>Vector database</b> that supports RAG algorithm and enhances the LLM agent<b>Software component</b> that interacts with the first two components to perform certain tasks<b>System</b> that manages resources and privileges to carry out the executionWhat's inside the artifact:For enhanced availability and reusability, we offer an organized defect library utilized in our manual studies.Below are details of what is included in each part:Application benchmarkA suite of 100 non-trivial projects which tightly integrates LLMs and vector databases in their workflow.We have uploaded application.csv, it contains:software project nameGitHub link and commit IDclassificationused LLM and vector databaseHydrangea Defect LibraryThe result of TABLE Ⅱ in our paper can be reproduced by this organized defect library.In the uploaded defect.csv, we have documented different cases for the same defect type, as defects can manifest in various ways. For each distinct case of the same defect, we have separated them with a blank line and labeled them as case 1, case 2, and so on, according to the specific circumstances.It contains:A collection of defects in these projects (involves 100 projects),containingthe defect type and its detailed explanationthe exact file and source-code line location of the defectthe consequences of defectthe defect-triggering testsThe meaning of different columns in defect.csv:<b>APP</b>: the applications from GitHub.<b>commit url</b>: the relevant version of the application on GitHub.<b>types</b>: different defect types.<b>cases</b>: different examples for each defect type. Cells containing a "/" indicate that there is only one case for that defect type.<b>explanation</b>: details of the defect.<b>consequences</b>: the impacts of the defect. Here we use the abbreviations: ST refer to fail-stops, IC refer to incorrectness, SL refer to slower execution, UI refer to unfriendly user interface, TK refer to more tokens, and IS refer to insecure.<b>source-code locations</b>: The location of the code file where the defect occurs.<b>defect-triggering tests</b>: The software input that triggers the defect.<br>