RAFT (Realworld Annotated Few-shot Tasks)

RAFT 基准（Realworld Annotated Few-shot Tasks）专注于自然发生的任务，并使用反映部署的评估设置。 RAFT 是一个测试语言模型的小样本分类基准：跨越多个领域（点燃的评论、医疗数据、推文、客户交互等），对具有经济价值的分类任务（有人天生关心该任务）进行评估，以反映部署（每个任务 50 个标记示例，允许信息检索，隐藏测试集）描述来自：https://raft.elicit.org/RAFT benchmark (Realworld Annotated Few-shot Tasks) focuses on naturally occurring tasks and adopts deployment-aligned evaluation settings. RAFT is a few-shot classification benchmark for testing language models: it evaluates economically significant classification tasks (tasks that people inherently care about) across multiple domains (e.g., literature reviews, medical data, tweets, customer interactions, etc.), with experimental setups that mirror real-world deployments, including 50 labeled examples per task, permission for information retrieval, and a hidden test set. The description is sourced from https://raft.elicit.org/

搜集汇总

背景与挑战

背景概述

RAFT是一个真实世界小样本文本分类基准，专注于自然发生的多领域任务，如评论和医疗数据，并通过模拟实际部署的评估设置来测试语言模型。该数据集由牛津大学等机构于2021年发布，旨在评估具有经济价值的分类任务。