Polygl0t/gigaverbo-v2-synth

---
dataset_info:
- config_name: default
features:
- name: text
dtype: string
- name: seed
dtype: string
- name: generator
dtype: string
- name: id
dtype: string
- name: token_count
dtype: int64
splits:
- name: train
num_bytes: 26565020943
num_examples: 11237546
download_size: 26565020943
dataset_size: 26565020943
configs:
- config_name: default
default: true
data_files:
- split: train
path: default/train-*
---

# GigaVerbo-v2 Synth: A Synthetic Dataset for Portuguese

<img src="./logo.png" height="200">

## Table of Contents

- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Subsets and Splits](#subsets-and-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Generation Pipeline](#generation-pipeline)
- [Quality and Filtering](#quality-and-filtering)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Additional Information](#additional-information)
- [Dataset Maintainers](#dataset-maintainers)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Acknowledgments](#acknowledgments)

## Dataset Description

- **Homepage:** https://huggingface.co/datasets/Polygl0t/gigaverbo-v2-synth
- **Repository:** https://huggingface.co/datasets/Polygl0t/gigaverbo-v2-synth
- **Point of Contact:** [Polygl0t](mailto:[e-mail])

### Dataset Summary

GigaVerbo-v2 Synth is a large synthetic Portuguese text corpus (~9.3 billion tokens) generated to complement the GigaVerbo-v2 web-sourced dataset. Inspired by approaches like Cosmopedia, this dataset was created to fill gaps in domains where web data is scarce or of lower quality, providing high-quality, diverse educational content. The dataset includes educational texts, tutorials, academic articles, blog posts, mathematical walkthroughs, programming tutorials, and synthetic conversations, designed specifically to enhance language model training for Portuguese.

### Supported Tasks and Leaderboards

This dataset can be utilized for tasks involving language modeling and training data augmentation for Portuguese NLP applications.

### Languages

Portuguese.

## Dataset Structure

### Data Instances

The dataset consists of synthetic text examples generated from diverse seeds and prompts. Each sample is uniquely identified and contains the generated text content.

### Data Fields

```json
{
"text": "A revolução da inteligência artificial (IA) tem sido um dos avanços tecnológicos mais significativos do século XXI...",
"seed": "https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus/tree/main/cosmopedia-v2",
"generator": "Qwen/Qwen2.5-32B-Instruct",
"id": "9a2c2f466c074fbd01420210962d17a9",
"token_count": 42
}
```

### Subsets and Splits

The dataset has a single split containing all 11,237,546 examples. However, users can create custom splits based on generator type or seed dataset for specific use cases.

```python
from datasets import load_dataset

# Load the main dataset
ds = load_dataset("Polygl0t/gigaverbo-v2-synth", "default", split="train")

# If you don't want to download the entire dataset, set streaming to `True`
ds = load_dataset("Polygl0t/gigaverbo-v2-synth", "default", split="train", streaming=True)
```

#### Statistics by Generator

| Generator | Examples | Total Tokens |
| ------------------------------------------------------------------------ | -------------- | ----------------- |
| [Qwen2.5-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct) | 7,719,118 | 6,484,418,687 |
| [Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct) | 1,806,691 | 1,625,788,204 |
| [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) | 1,711,737 | 1,211,544,076 |
| **Total** | **11,237,546** | **9,321,750,967** |

<details>
<summary><b>Tokens per Generator</b></summary>


</details>

#### Statistics by Seed Dataset

| Seed | Examples | Total Tokens |
| ---------------------------------------------------------------------------------------------------- | -------------- | ----------------- |
| [Cosmopedia-v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus/tree/main/cosmopedia-v2) | 1,896,692 | 1,669,702,784 |
| [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath) | 1,308,713 | 1,361,573,271 |
| [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) | 1,205,667 | 996,465,458 |
| [LegalPT](https://huggingface.co/datasets/eduagarcia/LegalPT_dedup) | 2,012,941 | 990,403,657 |
| [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) | 1,074,616 | 978,048,012 |
| [CodeParrot](https://huggingface.co/datasets/codeparrot/codeparrot-clean) | 512,331 | 785,518,858 |
| [Blogset BR](https://huggingface.co/datasets/thegoodfellas/blogset-br) | 820,862 | 667,773,253 |
| [StarCoder](https://huggingface.co/datasets/bigcode/starcoderdata) | 268,968 | 560,829,372 |
| [CAPES Theses](https://huggingface.co/datasets/eduagarcia/capes_teses_dissertacoes) | 340,792 | 371,964,656 |
| [BDTD](https://bdtd.ibict.br/vufind/) | 342,428 | 369,321,918 |
| [SciELO](https://huggingface.co/datasets/eduagarcia/scielo_abstracts) | 276,109 | 294,413,728 |
| [Historinhas](https://huggingface.co/datasets/Boakpe/historinhas) | 715,578 | 178,535,564 |
| [FinePersonas](https://huggingface.co/datasets/argilla/FinePersonas-Synthetic-Email-Conversations) | 443,729 | 85,060,998 |
| [Baixe Livros](https://www.baixelivros.com.br/dominio-publico/) | 16,370 | 9,712,263 |
| [Stanford Encyclopedia of Philosophy](https://plato.stanford.edu) | 1,750 | 2,427,175 |
| **Total** | **11,237,546** | **9,321,750,967** |

<details>
<summary><b>Tokens per Seed Dataset</b></summary>


</details>

#### ⭐ GigaVerbo-v2 Ablations: The Impact of 46B Tokens of Educational & Synthetic Data ⭐

We have conducted a small-scale ablation study to analyze the impact of our filtering pipeline / synthetic data generation on the performance of a language models trained on GigaVerbo-v2. Results show a significant improvement in downstream performance across benchmarks when compared to the first version of GigaVerbo.

All individual benchmark scores and their evolution across training time can be found in the [.plots](https://huggingface.co/datasets/Polygl0t/gigaverbo-v2-synth/tree/main/.plots) folder.


## Dataset Creation

### Curation Rationale

GigaVerbo-v2 Synth was created to complement the GigaVerbo-v2 web-sourced dataset by generating high-quality, diverse text data that fills gaps in domains where web data is scarce or of lower quality. Using a tiered approach with three Qwen2.5 models, we generated ~9.3 billion synthetic tokens across diverse domains including education, mathematics, programming, philosophy, and more.

### Source Data and Inspiration

The dataset was inspired by approaches like those documented in Cosmopedia. Using diverse seed datasets spanning education, science, technology, law, and literature to guide generation quality and topic coverage.

### Generation Pipeline

We employed three different Qwen2.5 models:

- **[Qwen2.5-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct)** - Complex tasks (e.g., detailed mathematical reasoning)
- **[Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct)** - Intermediate tasks and generation
- **[Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)** - Simpler tasks like summarization and rephrasing

All prompts used are available in the [ALL_PROMPTS](ALL_PROMPTS.md) file.

### Quality and Filtering

#### Benchmark Decontamination

We implemented a decontamination pipeline using contiguous n-gram matching (k-grams of length 8-32) to ensure our synthetic data did not contaminate evaluation benchmarks. Both exact and approximate matching modes were used to account for minor variations. All contaminated samples were removed.

#### Language Filtering

Given the multilingual nature of our models, we applied strict language filtering by removing samples with characters outside Portuguese Unicode ranges, while preserving numbers, punctuation, and symbols.

#### Computational Resources

Generated over January-June 2024 using 16 × NVIDIA A40 GPUs:

- **Total GPU Hours:** ~48,000
- **Energy Consumption:** ~14,400 kWh
- **Carbon Footprint:** ~5,328 kg CO₂e (5.3 metric tons)
- **Final Output:** 11,237,546 samples (~9.3B tokens)

## Considerations for Using the Data

### Social Impact of Dataset

This synthetic Portuguese corpus enhances training data availability for Portuguese language models, advancing NLP research applications for Portuguese.

## Additional Information

### Dataset Maintainers

- [Nicholas Kluge Corrêa](mailto:[e-mail])
- [Shiza Fatimah](mailto:[e-mail])
- [Aniket Sen](mailto:[e-mail])

### Licensing Information

The dataset is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).

### Citation Information

```latex
@misc{correa2026tucano2cool,
title={{Tucano 2 Cool: Better Open Source LLMs for Portuguese}},
author={Nicholas Kluge Corr{\^e}a and Aniket Sen and Shiza Fatimah and Sophia Falk and Lennard Landgraf and Julia Kastner and Lucie Flek},
year={2026},
eprint={2603.03543},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2603.03543},
}
```

### Acknowledgments

Polyglot is a project funded by the Federal Ministry of Education and Research (BMBF) and the Ministry of Culture and Science of the State of North Rhine-Westphalia (MWK) as part of TRA Sustainable Futures (University of Bonn) and the Excellence Strategy of the federal and state governments.

We also gratefully acknowledge the granted access to the [Marvin cluster](https://www.hpc.uni-bonn.de/en/systems/marvin) hosted by [University of Bonn](https://www.uni-bonn.de/en) along with the support provided by its High Performance Computing & Analytics Lab.

### Contributions

If you want to contribute, contact us at [[e-mail]](mailto:[e-mail])!