BSC-LT/m-personas

---
dataset_info:
features:
- name: id
dtype: string
- name: persona
dtype: string
- name: prompt
dtype: string
- name: messages
list:
- name: role
dtype: string
- name: content
dtype: string
- name: lang
dtype: string
- name: type
dtype: string
- name: theme
dtype: string
- name: parent_persona_id
dtype: string
- name: source_text
dtype: string
splits:
- name: id
num_bytes: 124627195
num_examples: 35580
- name: ja
num_bytes: 142335820
num_examples: 34279
- name: pt
num_bytes: 87799058
num_examples: 27517
- name: ar
num_bytes: 142497241
num_examples: 29458
- name: zh
num_bytes: 106108798
num_examples: 33474
- name: gl
num_bytes: 113450919
num_examples: 36564
- name: ru
num_bytes: 180661619
num_examples: 30641
- name: eu
num_bytes: 103709099
num_examples: 37854
- name: fr
num_bytes: 142640670
num_examples: 37013
- name: sw
num_bytes: 83559730
num_examples: 27876
- name: ca
num_bytes: 124544966
num_examples: 41499
- name: en
num_bytes: 134534855
num_examples: 38380
- name: ko
num_bytes: 122786061
num_examples: 31814
- name: es
num_bytes: 124366603
num_examples: 36094
- name: de
num_bytes: 109029619
num_examples: 32352
download_size: 1022450314
dataset_size: 1842652253
configs:
- config_name: default
data_files:
- split: id
path: data/id-*
- split: ja
path: data/ja-*
- split: pt
path: data/pt-*
- split: ar
path: data/ar-*
- split: zh
path: data/zh-*
- split: gl
path: data/gl-*
- split: ru
path: data/ru-*
- split: eu
path: data/eu-*
- split: fr
path: data/fr-*
- split: sw
path: data/sw-*
- split: ca
path: data/ca-*
- split: en
path: data/en-*
- split: ko
path: data/ko-*
- split: es
path: data/es-*
- split: de
path: data/de-*
license: apache-2.0
task_categories:
- question-answering
language:
- es
- eu
- ca
- ar
- en
- fr
- gl
- id
- ja
- ko
- pt
- ru
- sw
- zh
- de
size_categories:
- 100K<n<1M
---

# mPersonas: Multilingual Persona‑Driven Conversational Dataset

## Dataset Summary

**mPersonas** is a multilingual open-source dataset with high-quality persona descriptions synthetically generated by DeepSeek-V3–0324. It follows a *persona-driven data synthesis methodology*, similar to [PersonaHub](https://huggingface.co/datasets/proj-persona/PersonaHub).

- **Instances:** 510,000
- **Total tokens:** 173M
- 28M in personas
- 145M in conversations (105M in assistant turns)
- **Languages:** 15
- **License:** Apache 2.0

## Methodology

This section describes the step-by-step process for generating the **mPersonas** dataset.

<img src="images/diagram.png" alt="Data Generation Pipeline" width="800"/>

**Figure:** Data generation pipeline from seed documents to processed conversations. Similarity filtering is applied separately for each language.

### Seed Data

Personas are generated from a curated subset of documents from [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2), selected for their quality, extensive deduplication, and multilingual cultural diversity. Each language-specific persona is seeded exclusively from documents in the corresponding language subset, ensuring linguistic and cultural relevance.

### Persona Generation

We follow PersonaHub's approach, utilizing both text-to-persona and persona-to-persona generation methods. Personas vary in length and complexity to enhance dataset diversity. Persona-to-persona generation specifically helps surface uncommon but realistic personas. Generation prompts are language-specific, significantly increasing generation success rates compared to multilingual prompts.

### Persona Filtering

The post-processing pipeline includes three filtering stages:

- **Format validation**: Ensuring generated personas follow the required JSON structure.
- **Language detection**: Using [GlotLID](https://huggingface.co/cis-lmu/glotlid) to verify linguistic consistency between generated personas and seed texts. Personas failing this check or scoring below 0.5 in language detection are discarded.
- **Deduplication**:
- **N-gram Deduplication**: Minhash (shingle size = 1, similarity threshold = 0.7, 128 permutations) and LSH to remove near-duplicates efficiently.
- **Embedding-based Deduplication**: LaBSE embeddings with cosine similarity threshold set at 0.85 to detect and remove finer-grained duplicates within each language subset.

Persona filtering removed between 4% and 35% of generated content, depending on the language.

<img src="images/UMAP_all-data.png" alt="m-personas distribution" width="500"/>

**Figure:** UMAP projection of main topics generated on our dataset.

### Conversation Filtering

Final conversations undergo structural validation, including checks for:
- Correct format.
- Strict alternation between 'user' and 'assistant' roles.
- Beginning with 'user' and ending with 'assistant'.

## Supported Tasks and Use Cases

### Direct Uses

This dataset is a foundational resource designed to support the generation of diverse synthetic data. Researchers and developers can incorporate persona descriptions into prompts to enhance diversity in downstream synthetic datasets.

### Out-of-scope Use

While it can be used for LLM training or fine-tuning, the dataset was primarily designed to support diverse synthetic data generation, rather than to directly optimize model performance.

## Dataset Structure

Each example is structured as follows:

```json
{
"id": "<unique_alphanumerical_id>",
"persona": "<persona_description>",
"parent-persona-id": "...", // null if not using persona-to-persona
"prompt": "",
"messages": [
{ "role": "user", "content": "" },
{ "role": "assistant", "content": "" },
...
],
"lang": "<ISO639-code>",
"type": "<general/reasoning>",
"theme": "",
"source_text": "<original_fineweb_text>" // null if persona-to-persona
}
```

## How to Use

```python
from datasets import load_dataset

personas_es = load_dataset("BSC-LT/m-personas", "es")
```

## Bias, Risks, and Limitations

All datasets inevitably carry inherent biases, and this one is no exception. Given that it has been synthetically generated using a language model, its content is conditioned by the biases present in that generating model. However, by grounding these synthetic generations in seed data, we aim to mitigate additional biases by anchoring the outputs to the distribution and characteristics of the original base dataset, thus limiting the risk of compounding model-specific biases.

We report that upon superficial analysis, the personas generated present multiple societal and cultural biases, with strong gender and gender-occupation biases among the most prominent. Although it is deeply concerning, examining the resulting conversations, we see that these are not clearly transferred to the instruction dataset, alleviating the issue for downstream uses.

We highlight that our analyses of these biases are by no means exhaustive and fully acknowledge the inherent risks associated with releasing datasets that include biases. We urge researchers and developers to use it responsibly and take it into account, performing safety testing specific to their application.

## License

This dataset is released under the [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0).

## Acknowledgements
This work has been promoted and financed by the Ministerio para la Transformación Digital y de la Función Pública and Plan de Recuperación, Transformación y Resiliencia - Funded by EU – NextGenerationEU within the framework of the project Modelos del Lenguaje.

## Citation

```bibtex
@software{BSC-LT/m-personas,
author = {Da Dalt, Severino and Pikabea, Iñigo and Prats, Jaume and Pamies, Marc},
title = {mPersonas Dataset},
month = {July},
year = {2025},
url = {https://huggingface.co/datasets/BSC-LT/m-personas}
}
```