AllTheBacteria/BacCorpus100

---
license: apache-2.0
tags:
- pretraining
- genomics
- DNA
- Protein
- glm
- bacteria
- microbiology
pretty_name: BacCorpus100
size_categories:
- 1M<n<10M
---

# BacCorpus100

BacCorpus100 is a large-scale bacterial genome corpus for training and evaluating genome-aware genomic language models. It contains quality-controlled bacterial genomes represented at the genome level, with predicted protein-coding sequences stored as protein translations and intergenic regions stored as DNA sequences. Genomes were deduplicated at 100% identity using genome sketching.

BacCorpus100 spans approximately 7 million genomes, 20 billion protein-coding features, 16 billion intergenic regions, more than 150,000 species, and over 10,000 environments. The dataset is intended for large-scale pretraining and analysis of bacterial genomic language models.

## Dataset Description

- **Repository:** `AllTheBacteria/BacCorpus100`
- **Domain:** bacterial genomics
- **Data type:** genome-level bacterial sequence and annotation data
- **Primary modalities:** protein sequences and intergenic DNA
- **Organisms:** bacteria from isolate genomes and metagenome-assembled genomes
- **Scale:** approximately 7 million quality-controlled, deduplicated genomes
- **Primary use case:** pretraining and analysis of genomic language models in bacteria

BacCorpus was built by combining genomes from several public bacterial genome resources, including MGnify [1], SPIRE [2], HRGM [3], GTDB [4], mOTUs DB [5], and AllTheBacteria [6]. Genomes were uniformly quality controlled and annotated.

## Intended Uses

This dataset is intended for:

- pretraining bacterial genomic language models;
- studying bacterial genome organisation;
- building protein, DNA, mixed-modality, or genome-aware models;
- extracting protein or intergenic DNA sequences for representation learning;
- benchmarking model architectures that use genome context.

## How to Use

Because BacCorpus100 is large, we recommend streaming the dataset rather than downloading it locally.

```python
from datasets import load_dataset

ds = load_dataset(
"AllTheBacteria/BacCorpus100",
split="train",
streaming=True,
)

example = next(iter(ds))
print(example.keys())
```

## Schema

BacCorpus100 stores **one row per genome**. Each row contains aligned list-valued fields describing the features extracted from that genome.

| Column | Type | Description |
|---|---|---|
| `genome_id` | `string` | Genome identifier. |
| `contig_id` | `list<int64>` | Contig index for each feature. |
| `feature_id` | `list<int64>` | Feature index within a contig. |
| `molecule` | `list<string>` | Feature type: `AA` for translated CDS, `DNA` for intergenic region. |
| `start` | `list<int64>` | Feature start coordinate. |
| `end` | `list<int64>` | Feature end coordinate. |
| `strand` | `list<int64>` | Feature strand, typically `1` or `-1`. |
| `sequence` | `list<string>` | Protein or DNA sequence for the feature. |
| `source` | `string` | Source database or resource. |

The list-valued columns are aligned by position, so index `i` across `contig_id`, `feature_id`, `molecule`, `start`, `end`, `strand`, and `sequence` refers to the same genomic feature.

## Other resources
For deduplicated and clustered DNA and protein sequence datasets, see:

* Protein sequences: https://huggingface.co/datasets/AllTheBacteria/BacCorpus-prot-90
* Intergenic regions (DNA): https://huggingface.co/datasets/AllTheBacteria/BacCorpus-intergenic-dna-90

## Citation:
```bibtex
TBD
```

## References

[1] Richardson, L. et al. MGnify: the microbiome sequence data analysis resource in 2023. *Nucleic Acids Research* 51(D1), D753–D759 (2023). doi:10.1093/nar/gkac1080

[2] Schmidt, T. S. B. et al. SPIRE: a Searchable, Planetary-scale mIcrobiome REsource. *Nucleic Acids Research* 52(D1), D777–D783 (2024). doi:10.1093/nar/gkad943

[3] Almeida, A. et al. A unified catalog of 204,938 reference genomes from the human gut microbiome. *Nature Biotechnology* 39, 105–114 (2021). doi:10.1038/s41587-020-0603-3

[4] Parks, D. H. et al. GTDB: an ongoing census of bacterial and archaeal diversity through a phylogenetically consistent, rank normalized and complete genome-based taxonomy. *Nucleic Acids Research* 50(D1), D785–D794 (2022). doi:10.1093/nar/gkab776

[5] Dmitrijeva, M. et al. The mOTUs online database provides web-accessible genomic context to taxonomic profiling of microbial communities. *Nucleic Acids Research* 53(D1), D797–D805 (2025). doi:10.1093/nar/gkae1004

[6] Hunt, M. et al. AllTheBacteria – all bacterial genomes assembled, available, and searchable. *bioRxiv* (2024). doi:10.1101/2024.03.08.584059