varun-v-rao/squad

---
task_categories:
- question-answering
dataset_info:
features:
- name: id
dtype: string
- name: title
dtype: string
- name: context
dtype: string
- name: question
dtype: string
- name: answers
sequence:
- name: text
dtype: string
- name: answer_start
dtype: int32
splits:
- name: train
num_bytes: 79061690.62181075
num_examples: 87285
- name: validation
num_bytes: 10388764.166508988
num_examples: 10485
download_size: 16137496
dataset_size: 89450454.78831974
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: validation
path: data/validation-*
---

## Dataset Card for "squad"

This truncated dataset is derived from the Stanford Question Answering Dataset (SQuAD) for reading comprehension. Its primary aim is to extract instances from the original SQuAD dataset that align with the context length of BERT, RoBERTa, OPT, and T5 models.

### Preprocessing and Filtering

Preprocessing involves tokenization using the BertTokenizer (WordPiece), RoBertaTokenizer (Byte-level BPE), OPTTokenizer (Byte-Pair Encoding), and T5Tokenizer (Sentence Piece). Each sample is then checked to ensure that the length of the tokenized input is within the specified model_max_length for each tokenizer.

原始信息汇总

数据集概述

任务类别

• 问答系统

数据集信息

特征

• id: 字符串类型
• title: 字符串类型
• context: 字符串类型
• question: 字符串类型
• answers: 序列类型
  • text: 字符串类型
  • answer_start: 整数类型 (int32)

数据分割

• 训练集:
  • 字节数: 79061690.62181075
  • 样本数: 87285
• 验证集:
  • 字节数: 10388764.166508988
  • 样本数: 10485

数据大小

• 下载大小: 16137496 字节
• 数据集大小: 89450454.78831974 字节

配置

• 默认配置:
  • 训练集: data/train-*
  • 验证集: data/validation-*

预处理和过滤

• 使用 BertTokenizer (WordPiece), RoBertaTokenizer (Byte-level BPE), OPTTokenizer (Byte-Pair Encoding), 和 T5Tokenizer (Sentence Piece) 进行分词。
• 每个样本经过分词后，确保其长度在指定模型最大长度 (model_max_length) 范围内。