varun-v-rao/adversarial_hotpotqa

---
task_categories:
- question-answering
dataset_info:
features:
- name: question
dtype: string
- name: context
dtype: string
- name: id
dtype: string
- name: answers
struct:
- name: answer_start
sequence: int64
- name: text
sequence: string
splits:
- name: train
num_bytes: 89560671.51114564
num_examples: 33358
- name: validation
num_bytes: 7454710.584712826
num_examples: 2828
download_size: 17859339
dataset_size: 97015382.09585845
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: validation
path: data/validation-*
---

## Dataset Card for "squad"

This truncated dataset is derived from the Stanford Question Answering Dataset (SQuAD) for reading comprehension. Its primary aim is to extract instances from the original SQuAD dataset that align with the context length of BERT, RoBERTa, OPT, and T5 models.

### Preprocessing and Filtering

Preprocessing involves tokenization using the BertTokenizer (WordPiece), RoBertaTokenizer (Byte-level BPE), OPTTokenizer (Byte-Pair Encoding), and T5Tokenizer (Sentence Piece). Each sample is then checked to ensure that the length of the tokenized input is within the specified model_max_length for each tokenizer.

原始信息汇总

数据集概述

任务类别

• 问答系统

数据集信息

• 特征：

  • question：问题，数据类型为字符串。
  • context：上下文，数据类型为字符串。
  • id：标识符，数据类型为字符串。
  • answers：答案，结构如下：
    • answer_start：答案开始位置，数据类型为整数序列。
    • text：答案文本，数据类型为字符串序列。

• 拆分：

  • train：训练集，包含33358个样本，大小为89560671.51114564字节。
  • validation：验证集，包含2828个样本，大小为7454710.584712826字节。

• 数据大小：

  • 下载大小：17859339字节
  • 数据集大小：97015382.09585845字节

配置

• 默认配置：
  • 训练集文件路径：data/train-*
  • 验证集文件路径：data/validation-*

预处理和过滤

• 预处理包括使用BertTokenizer、RoBertaTokenizer、OPTTokenizer和T5Tokenizer进行分词。
• 每个样本经过分词后，确保其长度在指定模型最大长度（model_max_length）范围内。