Anon126/my-raft-submission

--- benchmark: raft type: prediction submission_name: none --- # RAFT submissions for my-raft-submission ## Submitting to the leaderboard To make a submission to the [leaderboard](https://huggingface.co/spaces/ought/raft-leaderboard), there are three main steps: 1. Generate predictions on the unlabeled test set of each task 2. Validate the predictions are compatible with the evaluation framework 3. Push the predictions to the Hub! See the instructions below for more details. ### Rules 1. To prevent overfitting to the public leaderboard, we only evaluate **one submission per week**. You can push predictions to the Hub as many times as you wish, but we will only evaluate the most recent commit in a given week. 2. Transfer or meta-learning using other datasets, including further pre-training on other corpora, is allowed. 3. Use of unlabeled test data is allowed, as is it always available in the applied setting. For example, further pre-training using the unlabeled data for a task would be permitted. 4. Systems may be augmented with information retrieved from the internet, e.g. via automated web searches. ### Submission file format For each task in RAFT, you should create a CSV file called `predictions.csv` with your model's predictions on the unlabeled test set. Each file should have exactly 2 columns: * ID (int) * Label (string) See the dummy predictions in the `data` folder for examples with the expected format. Here is a simple example that creates a majority-class baseline: ```python from pathlib import Path import pandas as pd from collections import Counter from datasets import load_dataset, get_dataset_config_names tasks = get_dataset_config_names("ought/raft") for task in tasks: # Load dataset raft_subset = load_dataset("ought/raft", task) # Compute majority class over training set counter = Counter(raft_subset["train"]["Label"]) majority_class = counter.most_common(1)[0][0] # Load predictions file preds = pd.read_csv(f"data/{task}/predictions.csv") # Convert label IDs to label names preds["Label"] = raft_subset["train"].features["Label"].int2str(majority_class) # Save predictions preds.to_csv(f"data/{task}/predictions.csv", index=False) ``` As you can see in the example, each `predictions.csv` file should be stored in the task's subfolder in `data` and at the end you should have something like the following: ``` data ├── ade_corpus_v2 │ ├── predictions.csv │ └── task.json ├── banking_77 │ ├── predictions.csv │ └── task.json ├── neurips_impact_statement_risks │ ├── predictions.csv │ └── task.json ├── one_stop_english │ ├── predictions.csv │ └── task.json ├── overruling │ ├── predictions.csv │ └── task.json ├── semiconductor_org_types │ ├── predictions.csv │ └── task.json ├── systematic_review_inclusion │ ├── predictions.csv │ └── task.json ├── tai_safety_research │ ├── predictions.csv │ └── task.json ├── terms_of_service │ ├── predictions.csv │ └── task.json ├── tweet_eval_hate │ ├── predictions.csv │ └── task.json └── twitter_complaints ├── predictions.csv └── task.json ``` ### Validate your submission To ensure that your submission files are correctly formatted, run the following command from the root of the repository: ``` python cli.py validate ``` If everything is correct, you should see the following message: ``` All submission files validated! ✨ 🚀 ✨ Now you can make a submission 🤗 ``` ### Push your submission to the Hugging Face Hub! The final step is to commit your files and push them to the Hub: ``` python cli.py submit ``` If there are no errors, you should see the following message: ``` Submission successful! 🎉 🥳 🎉 Your submission will be evaulated on Sunday 05 September 2021 ⏳ ``` where the evaluation is run every Sunday and your results will be visible on the leaderboard.--- 基准测试：RAFT 任务类型：预测 提交名称：无 --- # 面向my-raft-submission的RAFT基准测试提交指南 ## 向排行榜提交结果 若要向[排行榜](https://huggingface.co/spaces/ought/raft-leaderboard)提交结果，主要包含三个步骤： 1. 在每个任务的未标注测试集上生成模型预测结果 2. 验证预测结果与评估框架兼容 3. 将预测结果推送至Hugging Face Hub！ 详见下文说明。 ### 提交规则 1. 为避免对公开排行榜数据过拟合，我们仅**每周评估一次提交**。您可随时向Hugging Face Hub推送预测结果，但我们仅会评估当周内的最新提交记录。 2. 允许使用其他数据集进行迁移学习或元学习，包括在其他语料上进行进一步预训练。 3. 允许使用未标注测试数据，正如其在实际应用场景中始终可被获取一样。例如，针对某一任务使用对应未标注数据进行进一步预训练是被允许的。 4. 可通过互联网检索获取的信息（例如自动化网络搜索结果）对模型进行增强。 ### 提交文件格式 针对RAFT中的每个任务，您需要创建名为`predictions.csv`的CSV文件，存储模型在对应未标注测试集上的预测结果。每个文件需严格包含两列： * ID（整数类型） * Label（字符串类型） 可参考`data`文件夹中的示例预测文件，了解符合要求的文件格式。以下是一个构建多数类基准模型的简单示例： python from pathlib import Path import pandas as pd from collections import Counter from datasets import load_dataset, get_dataset_config_names tasks = get_dataset_config_names("ought/raft") for task in tasks: # Load dataset raft_subset = load_dataset("ought/raft", task) # Compute majority class over training set counter = Counter(raft_subset["train"]["Label"]) majority_class = counter.most_common(1)[0][0] # Load predictions file preds = pd.read_csv(f"data/{task}/predictions.csv") # Convert label IDs to label names preds["Label"] = raft_subset["train"].features["Label"].int2str(majority_class) # Save predictions preds.to_csv(f"data/{task}/predictions.csv", index=False) 如示例所示，每个`predictions.csv`文件需存储在`data`文件夹下对应任务的子文件夹中，最终的目录结构应如下所示： data ├── ade_corpus_v2 │ ├── predictions.csv │ └── task.json ├── banking_77 │ ├── predictions.csv │ └── task.json ├── neurips_impact_statement_risks │ ├── predictions.csv │ └── task.json ├── one_stop_english │ ├── predictions.csv │ └── task.json ├── overruling │ ├── predictions.csv │ └── task.json ├── semiconductor_org_types │ ├── predictions.csv │ └── task.json ├── systematic_review_inclusion │ ├── predictions.csv │ └── task.json ├── tai_safety_research │ ├── predictions.csv │ └── task.json ├── terms_of_service │ ├── predictions.csv │ └── task.json ├── tweet_eval_hate │ ├── predictions.csv │ └── task.json └── twitter_complaints ├── predictions.csv └── task.json ### 验证提交文件 为确保提交文件格式正确，请在仓库根目录执行以下命令： python cli.py validate 若格式无误，您将看到如下提示信息： All submission files validated! ✨ 🚀 ✨ Now you can make a submission 🤗 ### 将提交内容推送至Hugging Face Hub！ 最后一步是提交您的文件并将其推送至Hugging Face Hub： python cli.py submit 若推送无异常，您将看到如下提示信息： Submission successful! 🎉 🥳 🎉 Your submission will be evaluated on Sunday 05 September 2021 ⏳ 评估任务每周日执行，您的模型结果将随后在排行榜上展示。

原始信息汇总

RAFT 提交数据集

提交规则

1. 每周仅评估一次提交，以防止过度拟合公共排行榜。
2. 允许使用其他数据集进行迁移学习或元学习，包括在其他语料库上进一步预训练。
3. 允许使用未标记的测试数据，例如，使用任务的未标记数据进行进一步预训练是允许的。
4. 允许系统通过自动网络搜索等方式增强信息。

提交文件格式

对于RAFT中的每个任务，应创建一个名为predictions.csv的CSV文件，包含模型在未标记测试集上的预测。每个文件应包含以下两列：

• ID（整数）
• Label（字符串）

示例代码如下：

python from pathlib import Path import pandas as pd from collections import Counter from datasets import load_dataset, get_dataset_config_names

tasks = get_dataset_config_names("ought/raft")

for task in tasks: # 加载数据集 raft_subset = load_dataset("ought/raft", task) # 计算训练集上的多数类 counter = Counter(raft_subset["train"]["Label"]) majority_class = counter.most_common(1)[0][0] # 加载预测文件 preds = pd.read_csv(f"data/{task}/predictions.csv") # 将标签ID转换为标签名称 preds["Label"] = raft_subset["train"].features["Label"].int2str(majority_class) # 保存预测 preds.to_csv(f"data/{task}/predictions.csv", index=False)

每个predictions.csv文件应存储在任务的子文件夹中，最终结构如下：

data ├── ade_corpus_v2 │ ├── predictions.csv │ └── task.json ├── banking_77 │ ├── predictions.csv │ └── task.json ├── neurips_impact_statement_risks │ ├── predictions.csv │ └── task.json ├── one_stop_english │ ├── predictions.csv │ └── task.json ├── overruling │ ├── predictions.csv │ └── task.json ├── semiconductor_org_types │ ├── predictions.csv │ └── task.json ├── systematic_review_inclusion │ ├── predictions.csv │ └── task.json ├── tai_safety_research │ ├── predictions.csv │ └── task.json ├── terms_of_service │ ├── predictions.csv │ └── task.json ├── tweet_eval_hate │ ├── predictions.csv │ └── task.json └── twitter_complaints ├── predictions.csv └── task.json

验证提交

运行以下命令验证提交文件格式是否正确：

python cli.py validate

如果格式正确，将显示以下消息：

All submission files validated! ✨ 🚀 ✨ Now you can make a submission 🤗

提交到Hugging Face Hub

最后一步是提交文件到Hub：

python cli.py submit

如果没有错误，将显示以下消息：

Submission successful! 🎉 🥳 🎉 Your submission will be evaluated on Sunday 05 September 2021 ⏳

评估将在每周日进行，结果将在排行榜上可见。