ajsbsd/presto

---
license: cc-by-4.0
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
dataset_info:
features:
- name: inputs
dtype: string
- name: targets
dtype: string
- name: metadata
struct:
- name: locale
dtype: string
- name: example_id
dtype: string
- name: seeded_lists
list:
- name: name
dtype: string
- name: items
sequence: string
- name: seeded_notes
list:
- name: name
dtype: string
- name: content
dtype: string
- name: seeded_contacts
sequence: string
- name: previous_turns
list:
- name: user_query
dtype: string
- name: response_text
dtype: string
- name: linguistic_phenomena
dtype: string
- name: split
dtype: string
- name: context
dtype: string
splits:
- name: train
num_bytes: 24777921
num_examples: 33577
download_size: 6999588
dataset_size: 24777921
language:
- en
---

Code to test on Colab

!pip install -q transformers[torch] tokenizers datasets evaluate rouge_score sentencepiece huggingface_hub --upgrade

from huggingface_hub import notebook_login

notebook_login()

import nltk
from datasets import load_dataset
import evaluate
import numpy as np
from transformers import T5Tokenizer, DataCollatorForSeq2Seq
from transformers import T5ForConditionalGeneration, Seq2SeqTrainingArguments, Seq2SeqTrainer

# Load and split the dataset
dataset = load_dataset("ajsbsd/presto")
dataset = dataset["train"].train_test_split(test_size=0.2)
#dataset = load_dataset("csv", data_files="./JEOPARDY_CSV.csv")
#dataset = dataset["train"].train_test_split(test_size=0.2)
# Load the tokenizer, model, and data collator
tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-small")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-small")
data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)

# We prefix our tasks with "answer the question"
prefix = "answer the question: "

# Define our preprocessing function
def preprocess_function(examples):
"""Add prefix to the sentences, tokenize the text, and set the labels"""
# The "inputs" are the tokenized answer:
inputs = [prefix + doc for doc in examples["inputs"]]
model_inputs = tokenizer(inputs, max_length=128, truncation=True)

# The "labels" are the tokenized outputs:
labels = tokenizer(text_target=examples["targets"], max_length=512, truncation=True)
model_inputs["labels"] = labels["input_ids"]
return model_inputs

# Map the preprocessing function across our dataset
tokenized_dataset = dataset.map(preprocess_function, batched=True)

# Set up Rouge score for evaluation
nltk.download("punkt", quiet=True)
metric = evaluate.load("rouge")

def compute_metrics(eval_preds):
preds, labels = eval_preds

# decode preds and labels
labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

# rougeLSum expects newline after each sentence
decoded_preds = ["\n".join(nltk.sent_tokenize(pred.strip())) for pred in decoded_preds]
decoded_labels = ["\n".join(nltk.sent_tokenize(label.strip())) for label in decoded_labels]

result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
return result

# Set up training arguments
training_args = Seq2SeqTrainingArguments(
output_dir="./results",
evaluation_strategy="epoch",
learning_rate=3e-4,
per_device_train_batch_size=8,
per_device_eval_batch_size=4,
weight_decay=0.01,
save_total_limit=3,
num_train_epochs=2,
predict_with_generate=True,
push_to_hub=False
)

# Set up trainer
trainer = Seq2SeqTrainer(
model=model,
args=training_args,
train_dataset=tokenized_dataset["train"],
eval_dataset=tokenized_dataset["test"],
tokenizer=tokenizer,
data_collator=data_collator,
compute_metrics=compute_metrics
)

# Train the model
trainer.train()

# Push to HF :)
trainer.push_to_hub()

原始信息汇总

数据集概述

许可证

• 该数据集遵循CC BY 4.0许可证。

配置

• 默认配置（default）包含训练数据文件，路径为data/train-*。

数据集信息

• 特征：

  • inputs：输入文本，数据类型为字符串。
  • targets：目标文本，数据类型为字符串。
  • metadata：元数据，包含以下字段：
    • locale：地区信息，数据类型为字符串。
    • example_id：示例ID，数据类型为字符串。
    • seeded_lists：种子列表，包含名称和项目，数据类型为字符串列表。
    • seeded_notes：种子笔记，包含名称和内容，数据类型为字符串列表。
    • seeded_contacts：种子联系人，数据类型为字符串序列。
    • previous_turns：先前对话轮次，包含用户查询和响应文本，数据类型为字符串列表。
    • linguistic_phenomena：语言现象，数据类型为字符串。
    • split：数据分割，数据类型为字符串。
    • context：上下文，数据类型为字符串。

• 分割：

  • train：训练集，包含33577个样本，总字节数为24777921。

数据大小

• 下载大小：6999588字节。
• 数据集大小：24777921字节。

语言

• 该数据集主要包含英语（en）数据。