Shofo/shofo-tiktok-general-small

--- task_categories: - video-classification - text-generation - audio-classification language: - en - es tags: - short-form - video - transcripts - multimodal size_categories: - 10K<n<100K license: other --- # Shofo TikTok General (Small) ## Overview **Shofo TikTok General (Small)** is a dataset containing **50,000 TikTok videos** with comprehensive metadata, transcripts, comments, and engagement metrics. This is a curated subset of Shofo's larger TikTok index, which contains hundreds of millions of indexed videos. - **Size**: \~50K videos (\~500GB) - **Modality**: Video + Audio + Text (transcripts, comments, captions) - **Source**: TikTok ## Schema | Column | Type | Description | |--------|------|-------------| | `file_name` | string | Relative path to video file (e.g., `videos/123.mp4`) | | `video_id` | string | Unique TikTok video identifier | | `web_url` | string | TikTok web URL for the video | | `creator` | string | Creator username | | `transcript` | string | Audio transcription (ASR-generated, may be null) | | `description` | string | Video caption/description | | `hashtags` | JSON array | List of hashtags used | | `sticker_text` | JSON array | Text overlays/stickers visible in video | | `comments` | JSON array | Top comments with metadata (see below) | | `engagement_metrics` | JSON object | View counts, likes, shares, etc. (see below) | | `date_posted` | timestamp | When the video was originally posted | | `language` | JSON object | Language detection info (see below) | | `fps` | int | Frames per second | | `resolution` | string | Video resolution (e.g., `1080x1920`) | | `duration_ms` | int | Video duration in milliseconds | | `is_ai_generated` | bool | Whether the video was labeled as AI-generated | | `is_ad` | bool | Whether the video is an advertisement | ### Engagement Metrics Structure ```json { "play_count": 8948070, "like_count": 789584, "comment_count": 1451, "share_count": 38604, "collect_count": 126905, "repost_count": 0, "download_count": 235172, "whatsapp_share_count": 15737 } ``` ### Comments Structure Each comment in the `comments` array contains: ```json { "cid": "7352452026457342726", "text": "Comment text here", "create_time": 1711876158, "like_count": 885, "reply_count": 9, "username": "commenter_username", "user_region": "MX", "language": "es" } ``` ### Language Structure ```json { "desc_language": "es", "sticker_language": "en", "region": "US", "author_region": "US", "original_audio_language": null } ``` ## Collection Methodology Videos were collected through Shofo's TikTok indexing pipeline: 1. **Discovery**: Creators and hashtags are discovered through an explore/exploit strategy, snowballing from seed accounts 2. **Indexing**: Video metadata is fetched via TikTok's API 3. **Transcription**: Audio is transcribed using automatic speech recognition (ASR) 4. **Deduplication**: Videos are deduplicated using Redis-based ID tracking This subset represents a curated sample from the larger index, selected for data quality and diversity. ## Usage ### Using HuggingFace Datasets Library ```python from datasets import load_dataset ds = load_dataset("Shofo/shofo-tiktok-general-small", split="train") # Access a sample sample = ds[0] print(sample["transcript"]) print(sample["description"]) print(sample["engagement_metrics"]) ``` ### Using Pandas ```python import pandas as pd df = pd.read_parquet("hf://datasets/Shofo/shofo-tiktok-general-small/metadata.parquet") # Filter by engagement popular = df[df['engagement_metrics'].apply(lambda x: x['play_count'] > 1000000)] ``` ### Accessing Videos Videos are stored in the `videos/` directory and linked via the `file_name` column: ```python from datasets import load_dataset ds = load_dataset("Shofo/shofo-tiktok-general-small", split="train") # Get video path video_path = ds[0]["file_name"] # e.g., "videos/7350916080610643231.mp4" ``` ## Notes - **Compression**: Tiktok automatically uses H264 compression on its videos, achieving \~50x slightly lossy compression. - **Engagement metrics**: Values are from time of indexing - **Comments**: Top 50 comments at time of indexing - **Nulls**: Some fields may be null (e.g., `transcript` if no speech, `sticker_text` if no overlays) ## Larger Versions This is the "small" version of the Shofo TikTok dataset. Larger versions are available: - **Shofo TikTok General (Medium)**: 10m+ videos - **Shofo TikTok General (Large)**: 100M+ videos ## Citation ```bibtex @dataset{shofo_tiktok_general_small_2025, title={Shofo TikTok General (Small)}, author={Shofo}, year={2025}, url={https://huggingface.co/datasets/Shofo/shofo-tiktok-general-small} } ``` ## License & Disclaimer This dataset is provided for research and experimental use. Shofo does not claim ownership of the underlying video content. Users are responsible for ensuring compliance with applicable copyright laws and platform terms when using this dataset.任务类别： - 视频分类 - 文本生成 - 音频分类 语言： - 英语 - 西班牙语 标签： - 短视频 - 视频 - 转录文本 - 多模态 规模类别： - 10K<n<100K 许可证：其他 --- # Shofo TikTok通用（小型） ## 概述 **Shofo TikTok通用（小型）** 是一款包含50,000条TikTok视频的数据集，附带全面的元数据、音频转录文本、评论及互动指标。本数据集是Shofo旗下超大规模TikTok索引库的精选子集，该索引库收录了数亿条已索引的视频。 - 规模：约5万条视频（约500GB） - 模态：视频+音频+文本（转录文本、评论、字幕） - 来源：TikTok ## 数据结构 | 字段名 | 数据类型 | 描述 | |--------|------|-------------| | `file_name` | 字符串 | 视频文件的相对路径（例如：`videos/123.mp4`） | | `video_id` | 字符串 | TikTok视频的唯一标识符 | | `web_url` | 字符串 | 该视频的TikTok网页链接 | | `creator` | 字符串 | 创作者用户名 | | `transcript` | 字符串 | 音频转录文本（由自动语音识别（Automatic Speech Recognition, ASR）生成，可为空） | | `description` | 字符串 | 视频的字幕/描述文本 | | `hashtags` | JSON数组 | 所使用的话题标签列表 | | `sticker_text` | JSON数组 | 视频中可见的文字叠加层/贴纸文本 | | `comments` | JSON数组 | 带有元数据的热门评论（详见下文） | | `engagement_metrics` | JSON对象 | 播放量、点赞数、分享数等互动指标（详见下文） | | `date_posted` | 时间戳 | 视频最初发布的时间 | | `language` | JSON对象 | 语言检测相关信息（详见下文） | | `fps` | 整数 | 每秒帧数 | | `resolution` | 字符串 | 视频分辨率（例如：`1080x1920`） | | `duration_ms` | 整数 | 视频时长，单位为毫秒 | | `is_ai_generated` | 布尔值 | 该视频是否被标记为AI生成内容 | | `is_ad` | 布尔值 | 该视频是否为广告 | ### 互动指标结构 json { "play_count": 8948070, "like_count": 789584, "comment_count": 1451, "share_count": 38604, "collect_count": 126905, "repost_count": 0, "download_count": 235172, "whatsapp_share_count": 15737 } 各字段含义依次为：播放次数、点赞次数、评论次数、分享次数、收藏次数、转发次数、下载次数、WhatsApp分享次数。 ### 评论结构 `comments`数组中的每条评论包含以下字段： json { "cid": "7352452026457342726", "text": "评论文本示例", "create_time": 1711876158, "like_count": 885, "reply_count": 9, "username": "评论者用户名", "user_region": "MX", "language": "es" } 各字段含义依次为：评论唯一ID、评论文本、评论创建时间戳、该评论的点赞数、该评论的回复数、评论者用户名、评论者所在地区、评论使用的语言。 ### 语言信息结构 json { "desc_language": "es", "sticker_language": "en", "region": "US", "author_region": "US", "original_audio_language": null } 各字段含义依次为：视频描述文本的语言、贴纸/文字叠加层的语言、视频所在地区、创作者所在地区、原始音频的语言（可为空）。 ## 收集方法 视频通过Shofo的TikTok索引管道完成收集： 1. **发现阶段**：采用探索-利用策略挖掘创作者与话题标签，从种子账号出发进行滚雪球式扩展 2. **索引阶段**：通过TikTok官方API获取视频元数据 3. **转录阶段**：使用自动语音识别（ASR）对音频内容进行转录 4. **去重阶段**：基于Redis的ID追踪机制实现视频去重 本子集是从超大规模索引库中精选的样本，旨在保障数据质量与多样性。 ## 使用方法 ### 使用HuggingFace Datasets库 python from datasets import load_dataset ds = load_dataset("Shofo/shofo-tiktok-general-small", split="train") # 访问单条样本 sample = ds[0] print(sample["transcript"]) print(sample["description"]) print(sample["engagement_metrics"]) ### 使用Pandas python import pandas as pd df = pd.read_parquet("hf://datasets/Shofo/shofo-tiktok-general-small/metadata.parquet") # 根据互动指标筛选热门视频 popular = df[df['engagement_metrics'].apply(lambda x: x['play_count'] > 1000000)] ### 访问视频文件 视频存储在`videos/`目录下，可通过`file_name`字段关联获取对应路径： python from datasets import load_dataset ds = load_dataset("Shofo/shofo-tiktok-general-small", split="train") # 获取视频路径 video_path = ds[0]["file_name"] # 例如："videos/7350916080610643231.mp4" ## 补充说明 - **压缩格式**：TikTok对视频自动采用H264压缩格式，可实现约50倍的轻度有损压缩。 - **互动指标**：指标数值为索引时的实时快照数据 - **评论数据**：包含索引时刻的前50条热门评论 - **空值情况**：部分字段可能为空，例如无语音内容时`transcript`字段为空，无文字叠加层时`sticker_text`字段为空。 ## 更大规模版本 本数据集为Shofo TikTok数据集的「小型」版本，另有更大规模的版本可供使用： - **Shofo TikTok通用（中型）**：收录1000万+条视频 - **Shofo TikTok通用（大型）**：收录1亿+条视频 ## 引用格式 bibtex @dataset{shofo_tiktok_general_small_2025, title={Shofo TikTok通用（小型）}, author={Shofo}, year={2025}, url={https://huggingface.co/datasets/Shofo/shofo-tiktok-general-small} } ## 许可证与免责声明 本数据集仅用于研究与实验用途。 Shofo不主张对底层视频内容拥有所有权。 使用者需自行确保使用本数据集时符合适用的版权法律与平台服务条款。