jaiw/lex_fridman_podcast_embeddings

---
license: apache-2.0
task_categories:
- question-answering
---

# Description
This dataset contains csv's from the Lex Fridman podcast transcripts provided by [Whispering-GPT](https://huggingface.co/datasets/Whispering-GPT/lex-fridman-podcast).
I split the episode transcripts into parent and child chunks for use with RAG.
The parent chunks is size 500 and the child is size 50. The children come with embeddings using OpenAI `text-embedding-3-small` with 1024 dimensionality.

# Motivation
This was designed for use with [ParentDocumentRetriever](https://python.langchain.com/docs/modules/data_connection/retrievers/parent_document_retriever) or [LlamaIndex](https://docs.llamaindex.ai/en/stable/examples/node_postprocessor/MetadataReplacementDemo.html).
It should provide better retrievals for queries on specific details. Embedding all the chunks took a while so hopefully this saves someone time.
The size of the csv's allow for uploading with Supabase "Table Editor" because I didn't want to use postgres COPY.

原始信息汇总

数据集描述

该数据集包含来自Lex Fridman播客的转录文本，这些文本由Whispering-GPT提供。数据集将播客转录文本分割为父块和子块，用于RAG模型。父块大小为500，子块大小为50。子块包含使用OpenAI的text-embedding-3-small模型生成的嵌入向量，维度为1024。

动机

该数据集设计用于ParentDocumentRetriever或LlamaIndex，旨在提供针对特定细节查询的更好检索效果。嵌入所有块的过程耗时较长，希望该数据集能为用户节省时间。CSV文件的大小适合通过Supabase的“Table Editor”上传，避免了使用postgres COPY的需求。