AlpacaEval

Evaluation of instruction-following models (e.g., ChatGPT) typically requires human interactions. This is time-consuming, expensive, and hard to replicate. AlpacaEval in an LLM-based automatic evaluation that is fast, cheap, replicable, and validated against 20K human annotations. It is particularly useful for model development. Although we improved over prior automatic evaluation pipelines, there are still fundamental limitations like the preference for longer outputs. AlpacaEval provides the following:
Leaderboard: a leaderboard of common models on the AlpacaEval evaluation set. Caution: Automatic evaluator (e.g. GPT4) may be biased towards models that generate longer outputs and/or that were fine-tuned on the model underlying the evaluator (e.g. GPT4).
Automatic evaluator: an automatic evaluator that has high agreement with humans (validated on 20K annotations). We evaluate a model by measuring the fraction of times an powerful LLM (e.g. GPT 4 or Claude or ChatGPT) prefers the outputs from that model over outputs from a reference model. Our evaluators enable caching and output randomization by default.
Toolkit for building automatic evaluators: a simple interface for building advanced automatic evaluators (e.g. with caching, batching, or multi-annotators) and analyzing them (quality, price, speed, statistical power, bias, variance etc).
Human evaluation data: 20K human preferences between a given and reference model on the AlpacaFarm evaluation set. 2.5K of these are cross-annotations (4 humans annotating the same 650 examples).
AlpacaEval dataset: a simplification of AlpacaFarm's evaluation set, where "instructions" and " inputs" are merged into one field, and reference outputs are longer. Details here.

对指令跟随模型（如ChatGPT）的评估通常需要人类交互，此类方式耗时费力、成本高昂且难以复现。AlpacaEval是一款基于大语言模型（Large Language Model，LLM）的自动评估方案，兼具快速、低成本、可复现的优势，且经2万条人工标注数据验证，尤其适用于模型开发环节。尽管相较于此前的自动评估管线已有改进，但仍存在根本性局限，例如对更长输出文本的偏好。

AlpacaEval提供以下内容：
- 排行榜：AlpacaEval评估集上常见模型的性能排行榜。注意：自动评估器（如GPT-4）可能对生成更长输出的模型，以及在与评估器同源的模型（如GPT-4）上微调过的模型存在偏好性偏差。
- 自动评估器：与人类评估一致性较高的自动评估方案（经2万条标注数据验证）。我们通过衡量强大语言模型（如GPT-4、Claude或ChatGPT）相较于参考模型，更偏好待评估模型输出的比例，来完成模型评测。默认支持缓存与输出随机化功能。
- 自动评估器构建工具包：提供简洁接口，可用于构建高级自动评估器（如支持缓存、批量处理或多评估器协同），并可对评估方案的各项指标进行分析（包括质量、成本、速度、统计效力、偏差、方差等）。
- 人工评估数据：在AlpacaFarm评估集上，针对待评估模型与参考模型输出的2万条人类偏好标注数据，其中2.5千条为交叉标注数据（即4位人类标注员对同650个样本进行标注）。
- AlpacaEval数据集：对AlpacaFarm评估集的简化版本，将"指令"与"输入"合并为单个字段，且参考输出文本更长。详细信息见此处。

搜集汇总

背景与挑战

背景概述

AlpacaEval是一个用于评估指令跟随模型的自动评估数据集，基于大型语言模型实现快速、低成本且可复现的评测，并通过20K人类标注进行验证。它提供模型排行榜、评估工具包和人类偏好数据，主要用于模型开发，但需注意自动评估器可能对长输出或特定训练模型存在偏差。