yashsingh1/xdaforums

---
license: apache-2.0
---

# XDA Forums Dataset

<!-- Provide a quick summary of the dataset. -->

This dataset contains all of the posts throughout the history of XDA Forums.

## Dataset Details

### Dataset Description

The dataset contains the following:
- Threads
- Posts
- Users
- Categories/forums

The posts are provided in both HTML and markdown format. Some of the markdown formatted text won't be perfect and may appear in a XenForo specific markup format.

### Dataset Sources

This dataset is sourced from HTML [entries](https://xdaforums.com/sitemap.xml) on xdaforums.com and using the XDA Forums Android/Web APIs.

## Dataset Format

The dataset contains several partitioned Parquet files with about 68 million posts. Alternatively, you can also use the original RocksDB formatted database. Take a look at the `orm.ts` file for a reference on how the RocksDB database is structured.