Judge Builder

# Judge Builder

_Have you ever disagreed with your LLM Judge? Often, built-in judges and simple guidelines will work for some cases, but fall short in others where they may not understand your domain or edge cases in the judging criteria. Judge Builder addresses this challenge._

Judge Builder is deployed as a Databricks App in your workspace - think of the Databricks App as UI sugar over the existing methods already built into MLflow. Start with our built-in judges or create your own custom judge. Then align your judge with expert judgment by providing human feedback. Judges are registered as scorers in your experiment so you can easily use them in online and offline evaluations.

- Feedback Collection - Collect expert feedback via the review app or trace UIs
- Judge Alignment - Align judges with expert feedback using state-of-the-art optimization techniques
- Judge Lifecycle Management - From judge creation to production traffic evaluation, with a few lines of code
- MLflow Integration - Fully built on top of MLflow primitives and seamlessly integrates with your existing MLflow experiment - no migrations needed!

## Usage

1. **Create a Judge**: Start by creating a new judge with a name, instruction, and experiment ID. You will also invite SMEs to provide human feedback.
2. **Add Examples**: Add traces from the attached experiment to use as examples for your judge to learn from.
3. **Labeling**: The SME will provide human feedback over the added examples.
4. **Align Judge**: Run alignment to optimize the judge based on human feedback. You can review the performance of the judge before and after alignment.

You can also view a short video on Judge Builder [here](https://www.youtube.com/watch?v=OKnKtZj8f5Q).

## How to get help

For questions or bugs, please contact [e-mail] and the team will reach out shortly.

## License

© 2025 Databricks, Inc. All rights reserved. The source in this notebook is provided subject to the Databricks License [https://databricks.com/db-license-source].