nikhilchigali/wikianswers_embeddings_512

---
dataset_info:
features:
- name: sentence
dtype: string
- name: cluster
dtype: int64
- name: embedding_512
sequence: float32
splits:
- name: train
num_bytes: 2091790625
num_examples: 990526
download_size: 2669870589
dataset_size: 2091790625
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
task_categories:
- sentence-similarity
language:
- en
size_categories:
- 100K<n<1M
---

# Dataset Card for "wikianswers_embeddings_512"
## Dataset Summary
`nikhilchigali/wikianswers_embeddings_512` is a subset of the `embedding-data/WikiAnswers` ([Link](https://huggingface.co/datasets/embedding-data/WikiAnswers))
As opposed to the original dataset with 3,386,256 rows, this dataset contains only .13% of the total rows(sets). The sets of sentences have been unraveled into individual items with corresponding cluster IDs to identify sentences from the same set. Each Sentence has its associated cluster ID and embeddings of dimension 512.

## Languages
English.

## Dataset Structure
Each example in the dataset contains a sentence and its cluster id of other equivalent sentences. The sentences in the same cluster are paraphrases of each other. The embeddings for the dataset are created using the `distiluse-base-multilingual-cased-v1` model.
```
{"sentence": [sentence], "cluster": [cluster_id], "embedding_512": [embeddings]}
{"sentence": [sentence], "cluster": [cluster_id], "embedding_512": [embeddings]}
{"sentence": [sentence], "cluster": [cluster_id], "embedding_512": [embeddings]}
...
{"sentence": [sentence], "cluster": [cluster_id], "embedding_512": [embeddings]}
```
### Usage Example
Install the 🤗 Datasets library with `pip install datasets` and load the dataset from the Hub with:

```python
from datasets import load_dataset
dataset = load_dataset("nikhilchigali/wikianswers_embeddings_512")
```

The dataset is loaded as a DatasetDict and has the format for N examples:

```python
DatasetDict({
train: Dataset({
features: ['sentence', "cluster", "embedding_512"],
num_rows: N
})
})
```

Review an example i with:
```python
dataset["train"][i]
```
## Source Data
`embedding-data/WikiAnswers` on HuggingFace ([Link](https://huggingface.co/datasets/embedding-data/WikiAnswers))
### Note: This dataset is for the owner's personal use and claims no rights whatsoever.

原始信息汇总

数据集概述

数据集名称

nikhilchigali/wikianswers_embeddings_512

数据集来源

本数据集是embedding-data/WikiAnswers的一个子集，原始数据集包含3,386,256行，本数据集仅包含原始数据集的0.13%。

数据集结构

• 特征:
  • sentence: 字符串类型
  • cluster: 整数类型
  • embedding_512: 浮点数序列类型，维度512
• 数据分割:
  • train: 990526个样本，总大小2091790625字节
• 数据集大小:
  • 下载大小: 2669870589字节
  • 数据集大小: 2091790625字节

语言

英语

任务类别

句子相似性

数据集使用示例

python from datasets import load_dataset dataset = load_dataset("nikhilchigali/wikianswers_embeddings_512")

数据集加载格式

python DatasetDict({ train: Dataset({ features: [sentence, "cluster", "embedding_512"], num_rows: N }) })