filipwx/ted-podcast-finetune

--- license: cc-by-4.0 language: - en - pt tags: - ted-talks - podcasts - fine-tuning - llm - conversation - technology - leadership - business - science - lex-fridman - joe-rogan size_categories: - 1K<n<10K --- # LLM Fine-tuning Dataset: TED Talks + Podcasts A structured dataset of transcripts from popular TED Talks and podcasts (Lex Fridman Podcast, Joe Rogan Experience), formatted for LLM fine-tuning. ## Dataset Summary | Property | Value | |---|---| | **Total chunks** | 2,036 | | **Unique episodes/talks** | 48 | | **Train split** | 1,831 records | | **Validation split** | 204 records | | **Approx. total words** | 0 | | **Languages** | English (primary), Portuguese (some TED) | | **Format** | Chat / Instruction-response | ## Sources | Source | Chunks | Topics | |---|---|---| | TED Talks | 170 | Leadership, Technology, Business, Science, Psychology | | Lex Fridman Podcast | 1809 | AI, Technology, Science, Philosophy, Business | | Joe Rogan Experience | 57 | Technology, Science, Business, Society | ## Format ### OpenAI Fine-tuning (Chat format) Files: `dataset_train.jsonl`, `dataset_validation.jsonl` ```json { "messages": [ {"role": "system", "content": "You are a knowledgeable expert..."}, {"role": "user", "content": "What are the key ideas discussed here?"}, {"role": "assistant", "content": "The core argument is..."} ] } ``` ### Hugging Face Format File: `dataset_huggingface.jsonl` ```json { "id": "ted_qp0HIF3SfI4_chunk0_qa", "source": "TED", "language": "en", "category": "leadership", "type": "qa_pair", "text": "...", "instruction": "What is the main argument of this talk?", "system": "You are a knowledgeable assistant...", "messages": [...], "split": "train", "word_count": 512 } ``` ### Text Completion Format File: `dataset_text_completion.jsonl` ```json { "id": "ted_qp0HIF3SfI4_chunk0", "source": "TED", "title": "How great leaders inspire action", "language": "en", "text": "The core argument is..." } ``` ## Data Cleaning All transcripts were processed through: 1. **Timestamp removal**: `[00:01:23]`, `(00:01)`, `0:01:23` 2. **Speaker label removal**: `Lex Fridman:`, `SPEAKER_00:`, `Host:` 3. **Noise annotation removal**: `[applause]`, `[laughter]`, `(Music)` 4. **Boilerplate removal**: Podcast intros, sponsor messages, contact info 5. **URL/email removal** 6. **Unicode normalization**: Smart quotes → straight quotes 7. **Whitespace normalization** 8. **Minimum length filter**: Chunks with <30 words removed ## Usage ### Load with Hugging Face Datasets ```python from datasets import load_dataset dataset = load_dataset("json", data_files={ "train": "dataset_train.jsonl", "validation": "dataset_validation.jsonl" }) ``` ### OpenAI Fine-tuning ```python from openai import OpenAI client = OpenAI() with open("dataset_train.jsonl", "rb") as f: file = client.files.create(file=f, purpose="fine-tune") job = client.fine_tuning.jobs.create( training_file=file.id, model="gpt-4o-mini" ) ``` ### Validate before upload ```python import json with open("dataset_train.jsonl") as f: for i, line in enumerate(f): rec = json.loads(line) assert "messages" in rec for msg in rec["messages"]: assert "role" in msg and "content" in msg print(f"All {i+1} records valid!") ``` ## Topics Covered **TED Talks:** Leadership & Management, Motivation & Productivity, Psychology & Behavior, Technology & Innovation, Creativity & Education, Philosophy & Ethics, Science & Neuroscience, Communication **Lex Fridman Podcast:** Artificial Intelligence, Machine Learning, Software Engineering, Neuroscience, Physics & Mathematics, Geopolitics, Entrepreneurship **Joe Rogan Experience:** Technology, Science, Health & Fitness, Philosophy ## Files | File | Description | |---|---| | `dataset_train.jsonl` | Training split — OpenAI/HF chat format | | `dataset_validation.jsonl` | Validation split — OpenAI/HF chat format | | `dataset_huggingface.jsonl` | Full dataset with metadata | | `dataset_text_completion.jsonl` | Plain text completion format | | `dataset_full.csv` | CSV with all chunks | | `dataset_episodes.csv` | Episode-level summary | ## License This dataset is released under CC-BY 4.0. Transcripts are derived from publicly available content. TED transcripts © TED Conferences LLC (used for research/educational purposes). Podcast transcripts are from publicly available sources. ## Citation ``` @dataset{ted_podcast_finetune_2026, title={TED Talks + Podcasts LLM Fine-tuning Dataset}, year={2026}, publisher={Filipe Machado / Bit Pag LTDA}, sources={TED.com, lexfridman.com, joerogan.com}, format={JSONL / CSV}, records={2,036} } ```许可证：CC-BY-4.0 语言： - en（英语） - pt（葡萄牙语） 标签： - TED演讲（TED Talks） - 播客（podcasts） - 微调（fine-tuning） - 大语言模型（Large Language Model，LLM） - 对话（conversation） - 技术（technology） - 领导力（leadership） - 商业（business） - 科学（science） - 莱克斯·弗里德曼播客（Lex Fridman Podcast） - 乔·罗根体验秀（Joe Rogan Experience） 规模类别： - 1K < n < 10K # 大语言模型微调数据集：TED演讲 + 播客 本数据集为结构化的热门TED演讲与播客（莱克斯·弗里德曼播客、乔·罗根体验秀）转录文本集合，专为大语言模型微调设计。 ## 数据集概述 | 属性 | 取值 | |---|---| | **总片段数** | 2036 | | **唯一节目/演讲数量** | 48 | | **训练集划分** | 1831条记录 | | **验证集划分** | 204条记录 | | **近似总单词数** | 0 | | **语言** | 英语（主要语言），葡萄牙语（部分TED演讲） | | **格式** | 对话/指令-响应格式 | ## 数据来源 | 来源 | 片段数 | 主题 | |---|---| | TED演讲 | 170 | 领导力、技术、商业、科学、心理学 | | 莱克斯·弗里德曼播客 | 1809 | 人工智能、技术、科学、哲学、商业 | | 乔·罗根体验秀 | 57 | 技术、科学、商业、社会 | ## 格式 ### OpenAI 微调（对话格式） 文件：`dataset_train.jsonl`、`dataset_validation.jsonl` json { "messages": [ {"role": "system", "content": "You are a knowledgeable expert..."}, {"role": "user", "content": "What are the key ideas discussed here?"}, {"role": "assistant", "content": "The core argument is..."} ] } ### Hugging Face 格式 文件：`dataset_huggingface.jsonl` json { "id": "ted_qp0HIF3SfI4_chunk0_qa", "source": "TED", "language": "en", "category": "leadership", "type": "qa_pair", "text": "...", "instruction": "What is the main argument of this talk?", "system": "You are a knowledgeable assistant...", "messages": [...], "split": "train", "word_count": 512 } ### 文本补全格式 文件：`dataset_text_completion.jsonl` json { "id": "ted_qp0HIF3SfI4_chunk0", "source": "TED", "title": "How great leaders inspire action", "language": "en", "text": "The core argument is..." } ## 数据清洗 所有转录文本均经过以下步骤处理： 1. **移除时间戳**：移除形如`[00:01:23]`、`(00:01)`、`0:01:23`的时间标识 2. **移除说话人标识**：移除形如`Lex Fridman:`、`SPEAKER_00:`、`Host:`的说话人标注 3. **移除噪音标注**：移除形如`[applause]`、`[laughter]`、`(Music)`的环境噪音标注 4. **移除冗余模板内容**：移除播客开场、赞助信息、联系方式等模板化内容 5. **移除URL与邮箱地址** 6. **Unicode标准化处理**：将智能引号转换为直引号 7. **空格标准化处理** 8. **最小长度过滤**：移除单词量少于30的片段 ## 使用方法 ### 使用Hugging Face Datasets库加载 python from datasets import load_dataset dataset = load_dataset("json", data_files={ "train": "dataset_train.jsonl", "validation": "dataset_validation.jsonl" }) ### OpenAI 微调 python from openai import OpenAI client = OpenAI() with open("dataset_train.jsonl", "rb") as f: file = client.files.create(file=f, purpose="fine-tune") job = client.fine_tuning.jobs.create( training_file=file.id, model="gpt-4o-mini" ) ### 上传前校验 python import json with open("dataset_train.jsonl") as f: for i, line in enumerate(f): rec = json.loads(line) assert "messages" in rec for msg in rec["messages"]: assert "role" in msg and "content" in msg print(f"All {i+1} records valid!") ## 涵盖主题 **TED演讲涵盖主题：** 领导力与管理、激励与生产力、心理学与行为、技术与创新、创意与教育、哲学与伦理、科学与神经科学、沟通技巧 **莱克斯·弗里德曼播客涵盖主题：** 人工智能、机器学习、软件工程、神经科学、物理与数学、地缘政治、创业学 **乔·罗根体验秀涵盖主题：** 技术、科学、健康与健身、哲学 ## 文件清单 | 文件 | 描述 | |---|---| | `dataset_train.jsonl` | 训练集——支持OpenAI/Hugging Face对话格式 | | `dataset_validation.jsonl` | 验证集——支持OpenAI/Hugging Face对话格式 | | `dataset_huggingface.jsonl` | 包含完整元数据的全量数据集 | | `dataset_text_completion.jsonl` | 纯文本补全格式数据集 | | `dataset_full.csv` | 包含所有片段的CSV格式文件 | | `dataset_episodes.csv` | 节目级摘要CSV格式文件 | ## 许可证 本数据集采用CC-BY-4.0许可证发布。转录文本均源自公开可获取的内容。TED演讲转录文本© TED Conferences LLC（仅供研究与教育用途）。播客转录文本源自公开可获取的来源。 ## 引用 @dataset{ted_podcast_finetune_2026, title={TED Talks + Podcasts LLM Fine-tuning Dataset}, year={2026}, publisher={Filipe Machado / Bit Pag LTDA}, sources={TED.com, lexfridman.com, joerogan.com}, format={JSONL / CSV}, records={2,036} }