akarrouch-mohamed/bluesky

--- language: - en pretty_name: Curated English Bluesky Corpus tags: - bluesky - social-media - english - text-corpus - dataset-curation size_categories: - 10M<n<100M license: apache-2.0 --- # Curated English Bluesky Corpus ## Dataset summary This dataset is a large English-only Bluesky text corpus curated for tokenizer and language-model experiments. The corpus was built in two stages: 1. **Stage 1:** a full download of the Hugging Face dataset `Roronotalt/bluesky`, filtered to English posts and deduplicated by post URI. 2. **Stage 2:** additional English Bluesky posts collected directly from Bluesky repositories using a custom extractor, again deduplicated by URI and checked against Stage 1 to avoid overlap. The final merged corpus contains **90,831,330 rows** and **1,500,002,598 whitespace-counted words**. Only two columns are kept in the released dataset: - `uri`: the Bluesky AT URI of the post - `text`: the post text ## Why this dataset was created This corpus was curated to obtain a large English social-media fitting corpus with a target size of roughly **1.5 billion words**, suitable for tokenizer training and related downstream experiments. ## Data fields - `uri` (`string`): unique post identifier in AT URI format. - `text` (`string`): raw post text. ## Dataset splits The final release is stored as a `DatasetDict` with three splits: - **train:** 89,062,186 rows - **validation:** 891,793 rows - **test:** 877,351 rows Split creation was based on **cumulative whitespace word counts**, not random row sampling: - validation was filled first until it reached approximately **1%** of total words, - test was filled next until it reached approximately **1%** of total words, - train contains the remainder. Final split word counts are: - **train:** 1,470,002,529 words - **validation:** 15,000,031 words - **test:** 15,000,038 words ## Curation pipeline ### Stage 0: source download The initial source used for Stage 1 was the full `train` split of **`Roronotalt/bluesky`**, downloaded from Hugging Face in non-streaming mode and saved locally before filtering. ### Stage 1: filtering and deduplication of the Hugging Face source The downloaded source was processed row by row with the following rules: 1. Keep a row only if its `langs` metadata contains `en` or a value starting with `en-`. 2. Drop rows with missing or empty `uri`. 3. Drop rows with missing or empty `text`. 4. Count words using whitespace splitting and drop rows with fewer than 1 word. 5. Deduplicate by `uri`. 6. Keep only the columns `uri` and `text`. Stage 1 statistics: - **Processed source samples:** 94,967,071 - **English rows detected:** 60,854,982 - **Rows dropped for missing URI:** 0 - **Rows dropped for missing text:** 2,305,001 - **Rows dropped for fewer than 1 word:** 2,640 - **Rows dropped as duplicate URI:** 0 - **Final Stage 1 rows kept:** 58,547,341 - **Final Stage 1 words kept:** 966,013,061 ### Stage 2: direct Bluesky extraction to extend the corpus After Stage 1, a second collection pass was run to add more English Bluesky posts until the corpus reached the target scale. This step used a custom Go extractor with the following logic: 1. Export all Stage 1 URIs into a lookup file. 2. Query the Bluesky relay at `https://bsky.network` using `SyncListReposByCollection("app.bsky.feed.post")` to discover repositories containing posts. 3. For each discovered DID, resolve its PDS endpoint. 4. Fetch the repository snapshot with `SyncGetRepo`. 5. Stream repository records and keep only records whose path starts with `app.bsky.feed.post/`. 6. Read post text and `langs` metadata from each record. 7. Apply the same English rule as in Stage 1: keep posts only if `langs` contains `en` or a tag starting with `en-`. 8. Drop posts with missing text, missing URI, or fewer than 1 whitespace-counted word. 9. Drop posts whose URI already appeared in Stage 1. 10. Deduplicate within Stage 2 by URI. 11. Stop collection once the additional corpus exceeded the target number of new words needed to bring the merged corpus to roughly 1.5B words. Stage 2 statistics: - **Stage 1 URIs exported for duplicate checking:** 58,547,341 - **Repositories processed:** 8,977 - **Posts seen:** 51,955,104 - **English posts seen:** 33,219,791 - **Dropped for missing text:** 2,611,525 - **Dropped for non-English:** 16,123,788 - **Dropped for fewer than 1 word:** 13,837 - **Dropped for missing URI:** 0 - **Dropped as duplicate with Stage 1:** 921,965 - **Dropped as duplicate within Stage 2:** 0 - **Final Stage 2 rows kept:** 32,283,989 - **Final Stage 2 words kept:** 533,989,535 ### Final merge and split materialization The Stage 1 and Stage 2 datasets were concatenated, then deduplicated once more by `uri` during the final merge step. No additional duplicate URIs were found at this stage. Final merge statistics: - **Rows loaded from Stage 1:** 58,547,341 - **Rows loaded from Stage 2:** 32,283,989 - **Rows after concatenation:** 90,831,330 - **Rows dropped for empty URI during final merge:** 0 - **Rows dropped for empty text during final merge:** 0 - **Rows dropped as duplicate URI during final merge:** 0 - **Final total rows:** 90,831,330 - **Final total words:** 1,500,002,598 ## Curation rules at a glance - **Language filter:** keep only rows whose language metadata contains `en` or an `en-*` variant. - **Minimum length:** at least 1 whitespace-counted word. - **Deduplication:** URI-based. - **Released columns:** `uri`, `text`. - **Word counting:** whitespace-based (`split()` / `strings.Fields`). ## Important notes and limitations 1. **English detection relies on metadata, not an external language identification model.** In both stages, English selection was based on the `langs` field attached to the source record. 2. **Deduplication is URI-based, not text-based.** Near-duplicates or repeated text with different URIs may still remain. 3. **The validation and test splits are convenience splits.** They were created by sequential cumulative word count, not by time-based, user-based, or author-disjoint partitioning. 4. **Stage 2 extraction skipped unavailable repositories.** During direct extraction, some repositories were unavailable, deactivated, taken down, not found, or timed out. These failures were logged and skipped rather than retried indefinitely. 5. **The corpus is intended primarily as a large-scale text resource.** It is well suited for tokenizer fitting and corpus-level experimentation, but the provided validation/test splits should not automatically be treated as a benchmark design for all downstream tasks. ## Recommended use This dataset is especially appropriate for: - tokenizer training, - corpus statistics, - vocabulary analysis, - language-model pretraining or continued pretraining experiments, - studies of social-media language variation. ## Source attribution Stage 1 was derived from the Hugging Face dataset **`Roronotalt/bluesky`**. Stage 2 was collected directly from Bluesky repositories through a custom extraction pipeline over Bluesky infrastructure, with the same English filtering and URI-level deduplication rules applied. ## Acknowledgement If you use this dataset, please acknowledge both: 1. the original `Roronotalt/bluesky` source used for Stage 1, and 2. this curated release, which adds English filtering, URI-based deduplication, direct Bluesky augmentation, and train/validation/test split materialization.--- 语言： - 英语 展示名称：精选英语Bluesky语料库 标签： - bluesky - 社交媒体 - 英语 - 文本语料库 - 数据集精选 规模类别： - 1000万 < 数据量 < 1亿 许可证：Apache-2.0 --- # 精选英语Bluesky语料库 ## 数据集概览 本数据集为专为分词器（Tokenizer）与大语言模型（Large Language Model）实验打造的大型英语专属Bluesky文本语料库。 该语料库分两个阶段构建： 1. **阶段一**：完整下载Hugging Face数据集`Roronotalt/bluesky`，筛选出英语帖子，并按帖子URI进行去重。 2. **阶段二**：通过自定义提取器直接从Bluesky仓库采集额外的英语Bluesky帖子，同样按URI进行去重，并与阶段一的语料库进行比对以避免内容重叠。 最终合并后的语料库包含**90,831,330条数据**与**1,500,002,598个按空格计数的单词**。 发布的数据集中仅保留两列： - `uri`：帖子的Bluesky AT URI - `text`：帖子的原始文本 ## 数据集构建初衷 本语料库的构建目标是获取一个大型英语社交媒体适配语料库，目标规模约为**15亿个单词**，适用于分词器训练及相关下游实验。 ## 数据字段 - `uri`（字符串类型）：采用AT URI格式的唯一帖子标识符。 - `text`（字符串类型）：帖子的原始文本内容。 ## 数据集划分 最终发布的数据集以`DatasetDict`格式存储，包含三个划分： - **训练集（train）**：89,062,186条数据 - **验证集（validation）**：891,793条数据 - **测试集（test）**：877,351条数据 数据集划分并非通过随机采样，而是基于**累计空格计数单词数**进行构建： - 优先填充验证集，直至其单词数达到总单词数的约**1%** - 随后填充测试集，直至其单词数达到总单词数的约**1%** - 训练集包含剩余全部数据。 各划分的单词数如下： - **训练集**：1,470,002,529个单词 - **验证集**：15,000,031个单词 - **测试集**：15,000,038个单词 ## 精选流程 ### 阶段0：数据源下载 阶段一的初始数据源为**`Roronotalt/bluesky`**的完整`train`划分，以非流式模式从Hugging Face下载并本地存储，随后进行筛选处理。 ### 阶段一：Hugging Face数据源的筛选与去重 下载的数据源将按行进行以下处理规则： 1. 仅保留元数据`langs`中包含`en`或以`en-`开头的行。 2. 删除`uri`缺失或为空的行。 3. 删除`text`缺失或为空的行。 4. 通过空格拆分统计单词数，删除单词数少于1的行。 5. 按`uri`进行去重。 6. 仅保留`uri`与`text`两列。 阶段一统计数据： - **处理的源样本数**：94,967,071 - **检测到的英语行数量**：60,854,982 - **因URI缺失删除的行数**：0 - **因文本缺失删除的行数**：2,305,001 - **因单词数不足1个删除的行数**：2,640 - **因URI重复删除的行数**：0 - **阶段一最终保留行数**：58,547,341 - **阶段一最终保留单词数**：966,013,061 ### 阶段二：直接从Bluesky提取数据以扩展语料库 阶段一完成后，进行第二轮采集以补充更多英语Bluesky帖子，直至语料库达到目标规模。 本步骤使用自定义Go语言提取器，处理逻辑如下： 1. 将阶段一的所有URI导出为查找文件。 2. 通过`SyncListReposByCollection("app.bsky.feed.post")`向`https://bsky.network`的Bluesky中继节点发起查询，以发现包含帖子的仓库。 3. 为每个发现的去中心化标识符（DID）解析其个人数据服务器（PDS）端点。 4. 通过`SyncGetRepo`获取仓库快照。 5. 流式处理仓库记录，仅保留路径以`app.bsky.feed.post/`开头的记录。 6. 从每条记录中读取帖子文本与`langs`元数据。 7. 采用与阶段一相同的英语筛选规则：仅保留`langs`中包含`en`或以`en-`开头的帖子。 8. 删除文本缺失、URI缺失或按空格计数的单词数少于1的帖子。 9. 删除URI已存在于阶段一语料库中的帖子。 10. 在阶段二内部按URI进行去重。 11. 当新增语料的单词数达到使合并后语料库规模接近15亿单词的目标阈值时，停止采集。 阶段二统计数据： - **用于重复校验导出的阶段一URI数**：58,547,341 - **处理的仓库数**：8,977 - **采集到的帖子总数**：51,955,104 - **采集到的英语帖子数**：33,219,791 - **因文本缺失删除的帖子数**：2,611,525 - **因非英语筛选删除的帖子数**：16,123,788 - **因单词数不足1个删除的帖子数**：13,837 - **因URI缺失删除的帖子数**：0 - **因与阶段一语料库重复删除的帖子数**：921,965 - **因阶段二内部URI重复删除的帖子数**：0 - **阶段二最终保留行数**：32,283,989 - **阶段二最终保留单词数**：533,989,535 ### 最终合并与数据集划分生成 将阶段一与阶段二的数据集进行拼接，随后在最终合并步骤中再次按`uri`进行去重。本阶段未发现额外的重复URI。 最终合并统计数据： - **从阶段一导入的行数**：58,547,341 - **从阶段二导入的行数**：32,283,989 - **拼接后的总行数**：90,831,330 - **最终合并阶段因URI为空删除的行数**：0 - **最终合并阶段因文本为空删除的行数**：0 - **最终合并阶段因URI重复删除的行数**：0 - **最终总行数**：90,831,330 - **最终总单词数**：1,500,002,598 ## 精选规则概览 - **语言筛选**：仅保留元数据语言字段包含`en`或以`en-*`开头的行。 - **最小长度限制**：至少包含1个按空格计数的单词。 - **去重方式**：基于URI去重。 - **发布列名**：`uri`、`text`。 - **单词计数方式**：基于空格拆分（`split()` / `strings.Fields`）。 ## 重要说明与局限性 1. **英语筛选依赖元数据而非外部语言识别模型**：两个阶段的英语帖子筛选均基于源记录自带的`langs`字段。 2. **去重基于URI而非文本内容**：仍可能存在文本近似重复但URI不同的帖子。 3. **验证集与测试集为便捷性划分**：划分方式基于累计单词数顺序填充，而非基于时间、用户或作者去重的分区方式。 4. **阶段二提取过程中跳过了不可用的仓库**：直接提取过程中，部分仓库无法访问、已停用、已下架、未找到或超时。此类失败仅记录并跳过，未进行无限重试。 5. **本语料库主要作为大规模文本资源设计**：其适用于分词器适配与语料级实验，但提供的验证集与测试集不应直接作为所有下游任务的基准测试方案。 ## 推荐应用场景 本数据集尤其适用于： - 分词器训练 - 语料库统计分析 - 词汇分析 - 语言模型预训练或持续预训练实验 - 社交媒体语言变体研究 ## 数据源归因 阶段一的数据源自Hugging Face数据集**`Roronotalt/bluesky`**。 阶段二的数据通过自定义提取管道直接从Bluesky基础设施的仓库采集，采用与阶段一相同的英语筛选与基于URI的去重规则。 ## 致谢 若您使用本数据集，请同时注明以下两项： 1. 阶段一使用的原始`Roronotalt/bluesky`数据源 2. 本次精选发布版本，其新增了英语筛选、基于URI的去重、直接Bluesky数据增补以及训练/验证/测试集划分生成功能。