nachoyawn/three-million-bluesky

该数据集最初包含了大约500万条帖子，但这些帖子中存在大量重复。经过去重处理后，数据集被整理为260万条帖子，并存储在`final_posts.jsonl`文件中。数据文件夹中的其他`.jsonl`文件未经检查，可能包含重复的帖子，用户可以根据需要自行去重。整个数据集大约包含300万条唯一的帖子。

This dataset contains approximately 3 million posts from the bluesky platform. The original dataset `raw.zip` contained about 5 million posts, but it was full of duplicates. After removing duplicates, the final `final_posts.jsonl` file contains about 2.6 million unique posts. The other `.jsonl` files in the data folder, which are not inside `raw.zip`, have not been checked for duplicates and may contain duplicates, but they are expected to contain around 3 million unique posts in total.