honvoh-school/youtube-tourism-sentiment

--- license: cc-by-nc-4.0 task_categories: - text-classification language: - en - fr - tr - es - de - pt tags: - youtube - sentiment - tourism - multilingual - africa - latam pretty_name: YouTube Tourism Sentiment (multilingual) size_categories: - 10K<n<100K --- # YouTube Tourism Sentiment (multilingual) YouTube comments from tourism-related videos across Africa, Latin America, and Turkey, cleaned and pre-labeled with `cardiffnlp/twitter-xlm-roberta-base-sentiment`. Built for a school LLM project (Farel Honvoh / Elif Oksuzali). ## Contents - **comments.parquet** — one row per comment, joined with video and channel metadata - **videos.parquet** — video-level metadata with `algorithm_score = view_count / subscriber_count` ## v2 snapshot - **20,650 comments** across **26 videos** from **18 channels** - Scope: tourism content in Africa (Benin, Egypt, Nigeria, etc.), Latin America (Colombia, Brazil, Venezuela, Peru), and Turkey - Labels generated by a multilingual pre-trained model (no human labeling) ## Schema — comments.parquet | Column | Type | Description | |---|---|---| | comment_id | str | Truncated SHA1 of (video_id + author + text[:100]) | | video_id | str | YouTube video ID | | channel_id | str | Channel ID | | channel_name | str | Channel name | | video_title | str | Video title | | algorithm_score | float | views / subscribers (proxy for video over-performance) | | author | str | Comment author username | | text_raw | str | Original text (emojis preserved) | | text_clean | str | Cleaned text (URLs and mentions stripped, abbreviations expanded) | | lang | str | ISO language code (langdetect) | | likes | int | Like count | | reply_count | int | Reply count | | published_at | str | Publication date | | is_reply | int | 1 if reply to another comment, 0 otherwise | | sentiment_label | str | positive / neutral / negative | | sentiment_score | float | Max softmax confidence | | themes | str | Detected themes (CSV): hospitality, landscape, culture, food, safety, price, infrastructure | ## Schema — videos.parquet | Column | Type | Description | |---|---|---| | video_id | str | YouTube video ID | | channel_id | str | Channel ID | | channel_name | str | Channel name | | subscriber_count | int | Channel subscriber count at scrape time | | title | str | Video title | | description | str | First 300 chars of video description | | published_at | str | Publication date | | view_count | int | Views at scrape time | | like_count | int | Likes at scrape time | | comment_count | int | Comment count at scrape time | | algorithm_score | float | view_count / subscriber_count | | url | str | YouTube URL | ## v2 distribution - **Sentiment**: 11,141 positive / 5,639 negative / 3,870 neutral - **Dominant languages**: EN, FR, PT, ES, TR — see `comments.parquet` for the full breakdown ## Regions covered | Region | Videos | Approx. comments | |---|---|---| | Benin (v1) | 11 | ~6,000 | | Africa (Egypt, Nigeria, Morocco, etc.) | 7 | ~5,500 | | Latin America (Colombia, Brazil, Venezuela, Peru) | 6 | ~5,500 | | Turkey | 2 | ~1,800 | ## Usage ```python import pandas as pd comments = pd.read_parquet("comments.parquet") videos = pd.read_parquet("videos.parquet") # Example: positive comments about food food_pos = comments[ (comments.sentiment_label == "positive") & (comments.themes.str.contains("food", na=False)) ] ``` From the Hugging Face Hub: ```python from huggingface_hub import hf_hub_download import pandas as pd path = hf_hub_download( repo_id="honvoh-school/youtube-tourism-sentiment", filename="comments.parquet", repo_type="dataset", ) df = pd.read_parquet(path) ``` ## Preprocessing applied - Unicode NFKC normalization - URL, @mention, and zero-width character stripping - Abbreviation expansion (btw → by the way, omg → oh my god, etc.) - **Emojis kept** (sentiment signal) - Language detection via `langdetect` - Deduplication on same-author + same-text (copy-paste spam) ## Source Scraped via the official **YouTube Data API v3** (`commentThreads.list` + `videos.list` + `channels.list`). Free within the default 10,000 units/day quota. Switched from Apify in v2 to stay within budget. Pipeline code: https://github.com/lifestyleentrepreneur/youtube-tourism-sentiment (private) ## Version history - **v1** (2026-04-17): 6,355 comments on 11 Benin tourism videos - **v2** (2026-04-19): +14,295 comments from Africa / LatAm / Turkey videos → 20,650 total ## License Comments are original user-generated content on YouTube and remain the property of their authors. This dataset is a derived work provided for **non-commercial academic research only** (CC BY-NC 4.0). Do not redistribute or use for commercial purposes.license: CC BY-NC 4.0 task_categories: - 文本分类 language: - 英语 - 法语 - 土耳其语 - 西班牙语 - 德语 - 葡萄牙语 tags: - YouTube - 情感分析 - 旅游 - 多语言 - 非洲 - 拉美 pretty_name: YouTube旅游情感（多语言版） size_categories: - 10K<n<100K --- # YouTube旅游情感（多语言版） 本数据集包含来自非洲、拉丁美洲及土耳其旅游相关视频的YouTube评论，已完成清洗并使用`cardiffnlp/twitter-xlm-roberta-base-sentiment`模型完成预标注。本数据集为一次高校大语言模型（Large Language Model, LLM）课程项目所构建（作者：Farel Honvoh、Elif Oksuzali）。 ## 数据集内容 - **comments.parquet**：每条评论占一行，已关联视频与频道元数据 - **videos.parquet**：视频级元数据，包含`algorithm_score = 观看量 / 订阅者数`指标 ## v2版本快照 - 共包含**20650条评论**，覆盖**26个视频**与**18个频道** - 覆盖范围：非洲（贝宁、埃及、尼日利亚等）、拉丁美洲（哥伦比亚、巴西、委内瑞拉、秘鲁）及土耳其的旅游相关内容 - 情感标签由多语言预训练模型生成，未经过人工标注 ## 数据结构 — comments.parquet | 列名 | 数据类型 | 描述 | |---|---|---| | comment_id | str | 由`video_id + 作者 + 文本前100字符`生成的截断SHA1哈希值 | | video_id | str | YouTube视频ID | | channel_id | str | 频道ID | | channel_name | str | 频道名称 | | video_title | str | 视频标题 | | algorithm_score | float | 观看量/订阅者数（视频表现超额代理指标） | | author | str | 评论作者用户名 | | text_raw | str | 原始文本（保留表情符号） | | text_clean | str | 清洗后文本（移除URL与@提及内容，扩展缩写） | | lang | str | ISO语言代码（由`langdetect`检测） | | likes | int | 点赞数 | | reply_count | int | 回复数 | | published_at | str | 发布日期 | | is_reply | int | 1表示为其他评论的回复，0表示非回复 | | sentiment_label | str | 情感标签：正面/中性/负面 | | sentiment_score | float | 最大softmax置信度 | | themes | str | 检测到的主题（CSV格式）：住宿服务、自然风光、文化、美食、安全、价格、基础设施 | ## 数据结构 — videos.parquet | 列名 | 数据类型 | 描述 | |---|---|---| | video_id | str | YouTube视频ID | | channel_id | str | 频道ID | | channel_name | str | 频道名称 | | subscriber_count | int | 爬取时的频道订阅者数 | | title | str | 视频标题 | | description | str | 视频描述的前300个字符 | | published_at | str | 发布日期 | | view_count | int | 爬取时的观看量 | | like_count | int | 爬取时的点赞数 | | comment_count | int | 爬取时的评论数 | | algorithm_score | float | 观看量/订阅者数 | | url | str | YouTube视频链接 | ## v2版本分布 - **情感分布**：11141条正面评论、5639条负面评论与3870条中性评论 - **主要语言**：英语、法语、葡萄牙语、西班牙语、土耳其语，完整语言分布请查阅`comments.parquet`文件 ## 覆盖区域 | 区域 | 视频数量 | 评论数估算 | |---|---|---| | 贝宁（v1） | 11 | ~6000 | | 非洲（埃及、尼日利亚、摩洛哥等） |7 | ~5500 | | 拉丁美洲（哥伦比亚、巴西、委内瑞拉、秘鲁） |6 | ~5500 | | 土耳其 |2 | ~1800 | ## 使用方法 python import pandas as pd comments = pd.read_parquet("comments.parquet") videos = pd.read_parquet("videos.parquet") # 示例：筛选关于美食的正面评论 food_pos = comments[ (comments.sentiment_label == "positive") & (comments.themes.str.contains("food", na=False)) ] 从Hugging Face Hub加载数据集： python from huggingface_hub import hf_hub_download import pandas as pd path = hf_hub_download( repo_id="honvoh-school/youtube-tourism-sentiment", filename="comments.parquet", repo_type="dataset", ) df = pd.read_parquet(path) ## 预处理步骤 - Unicode NFKC标准化 - 移除URL、@提及内容与零宽字符 - 缩写扩展（例如`btw`扩展为`by the way`，`omg`扩展为`oh my god`等） - **保留表情符号**（可作为情感信号） - 通过`langdetect`进行语言检测 - 基于同一作者+同一文本进行去重（去除复制粘贴的垃圾评论） ## 数据来源 本数据集通过官方**YouTube Data API v3**（`commentThreads.list` + `videos.list` + `channels.list`）爬取获取，默认每日10000单位配额内免费使用。v2版本从Apify切换至该API以控制预算。 数据集处理流水线代码：https://github.com/lifestyleentrepreneur/youtube-tourism-sentiment（私有仓库） ## 版本历史 - **v1**（2026年4月17日）：覆盖11个贝宁旅游视频，共6355条评论 - **v2**（2026年4月19日）：新增来自非洲、拉美及土耳其视频的14295条评论，总评论数达20650条 ## 许可协议 本数据集包含的评论为YouTube平台上的原创用户生成内容，版权归原作者所有。本数据集为衍生作品，仅可用于**非商业学术研究**（许可协议：CC BY-NC 4.0）。严禁重新分发或用于商业用途。