madrylab/platinum-bench

Platinum Benchmarks是一组经过精心策划的基准测试，旨在最小化标签错误和歧义，以便测量模型的可靠性。该数据集包含十五个经过手动修订的基准测试，这些测试是从现有数据集中修订而来的。修订过程中，对于至少一个模型出错的示例进行了手动重新注释。数据集结构包括问题提示、目标答案等字段，并且还包含原始数据集的字段。为了使用修订后的基准测试，需要过滤掉被拒绝的问题。

Platinum Benchmarks are a set of carefully curated benchmarks designed to minimize label errors and ambiguity, allowing for the measurement of model reliability. This dataset contains fifteen manually revised benchmarks, which are revised from existing datasets. The revision process involves manually re-annotating examples that were incorrectly labeled by at least one model. The dataset structure includes fields such as question prompts, target answers, and also includes fields from the original dataset. To use the revised benchmarks, it is necessary to filter out rejected questions.