Better LLMs With Better Data Using Cleanlab Studio

https://www.databricks.com/solutions/accelerators/llm-cleanlab-studio

Data powers AI in the enterprise, but real-world data sets have been found to contain 7%–50% annotation errors. Unsurprisingly, erroneous data — from imperfectly labeled data to outliers — hampers the training (and evaluation) of ML models across tasks like intent recognition, entity recognition and sequence generation. Our joint Solution Accelerator with Cleanlab helps to improve training data to boost LLM performance by 37% without spending any time or resources to change the model architecture, hyperparameters or the training process.

* Fine-tune state-of-the-art LLMs using OpenAI’s APIs
* Evaluate trained LLMs to achieve high test accuracy (65%)
* Improve data quality using confident learning and DCAI without writing any code or having any ML expertise using Cleanlab Studio

Click on the "Get instant access" button in the top right corner to clone the solution accelerator repo into your workspace. Once the repo is cloned into your workspace, please execute the **RUNME** notebook in the repo in order to create the cluster and job you can use to run the notebooks.