Hydrangea

Hydrangea is a defect library for LLM-enabled software. Hydrangea has 4 main petals, each corresponding to one of the major components where defects often arise: LLM agent, vector database, software component, and system.What is LLM-enabled software?It is software that integrates LLMs (large language models) with RAG (retrieval-augmented generation ) support to realize intelligence features.It contains four components:<b>LLM agent</b> that manages LLM interfaces, constructs prompts, and invokes the LLM<b>Vector database</b> that supports RAG algorithm and enhances the LLM agent<b>Software component</b> that interacts with the first two components to perform certain tasks<b>System</b> that manages resources and privileges to carry out the executionWhat's inside the artifact:For enhanced availability and reusability, we offer an organized defect library utilized in our manual studies.Below are details of what is included in each part:Application benchmarkA suite of 100 non-trivial projects which tightly integrates LLMs and vector databases in their workflow.We have uploaded <code>application.csv</code>, it contains:software project nameGitHub link and commit IDclassificationused LLM and vector databaseHydrangea Defect LibraryThe result of TABLE Ⅱ in our paper can be reproduced by this organized defect library.In the uploaded <code>defect.csv</code>, we have documented different cases for the same defect type, as defects can manifest in various ways. For each distinct case of the same defect, we have separated them with a blank line and labeled them as case 1, case 2, and so on, according to the specific circumstances.It contains:A collection of defects in these projects (involves 100 projects),containingthe defect type and its detailed explanationthe exact file and source-code line location of the defectthe consequences of defectthe defect-triggering tests