Pile-OpenSubtitles

displayName: Pile-OpenSubtitles
license:
- MIT
taskTypes:
- Natural Language Generation
- Language Modelling
mediaTypes:
- Text
labelTypes:
- English Corpus
tags: []
publisher:
- EleutherAI
publishDate: '2023-07-18'
publishUrl: https://pile.eleuther.ai/
paperUrl: ''

---
# 数据介绍
## 简介
Pile-OpenSubtitles数据集是The Pile项目的一部分，是一个用于语言模型的数据集。该数据集基于OpenSubtitles项目的字幕文本构建而成。
OpenSubtitles是一个开源的字幕资源库，收集了大量的电影和电视剧的字幕文件。Pile-OpenSubtitles数据集利用这些字幕文本，为语言模型提供了丰富的对话和对话式文本资源。
Pile-OpenSubtitles数据集包含了各种类型的对话，涵盖了不同的主题和语言风格。它包括电影和电视剧中的对话，涵盖了从日常交流到科技讨论的广泛内容。
通过使用Pile-OpenSubtitles数据集，研究人员和开发者可以训练语言模型来理解和生成对话式的文本，从而在对话生成、情感分析、对话系统等领域应用中发挥作用。
## 数据内容
### 数据说明
Pile-OpenSubtitles数据集涵盖了6.7G的数据。
### 数据示例
```
{
"id": "146452926",
"source_id": "",
"doc_id": "27239921",
"data_type": "text",
"data_source": "pile",
"data_url": "enwiki-c4-pile-ccnews",
"content": "\"We're breaking up.\" \"It is not working out between us.\" \" We're breaking up.\" \" Not here.\" \"Not in front of the kids.\" \" We're breaking up.\" \" I can tell.\" \"I'm a doctor.\" \" You know we're breaking up, right?\" \" Oh, yeah.\" \" I have to break up with both of you.\" \" What?\" \" You know we're breaking up, right?\" \" Oh, yeah.\" \"It's not working for me.\" \"It's over.\" \"Again?\" \"It's over.\" \"Again?\" \"Okay, don't be so dramatic.\" \"We're breaking up.\" \"We're break\" \"Okay.\" \"I'm sorry.\" \"I've forgotten your name.\" \"We're breaking up!\" \"I can't hear you!\" \"No, I know that you can hear me 'cause I can hear you.\" \"We're breaking up.\" \"Okay.\" \"It's over.\" \"Will you e-mail me because I might forget.\" \"Things haven't been working out, and I just \" \"I'm breaking up with you.\" \"Beat you to it.\" \"Boom!\" \"We're breaking up.\" \"Huh?\" \"Can't it wait 20 seconds?\" \"10 seconds.\" \"No.\" \"I got to go.\" \"Aah!\" \"Wait!\" \"Attention, staff.\" \"There's a gigantic snake moving below the elevators.\" \"Love.\" \"Why is it that when you're in\" \"It, you can't remember not\" \"Being in it?\" \"And when you're not in it, it's\" \"Like, \"what is it?\"\" \"Why is it not just in and of\" \"Itself, in or out?\" \"I should do my own podcast.\" \"This looks like\" \"Esperanza the maid.\" \"well, it's like a big maid\" \"And a little maid.\" \"I don't like chocolate people.\" \"Oh, don't say that in public, honey.\" \"Hey.\" \"Dr. Black, you spend a lot of time treating Nicky, but his advanced-aging disease seems to be getting worse.\" \"I have gout.\" \"Why don't you go on down to the cafeteria and get us an eight-pack of smirnoff ice, yeah?\" \"Can you get on with the exam, please, before Nicky gets any older?\" \"God, look at those eyes.\" \"I have heat vision!\" \"God, I just wish I knew you when you were young.\" \"He's 6.\" \"Well, then I wish I knew you when you were 4.\" \"Here's a question.\" \"What?\" \"Are space bugs real?\" \"Do you like new experiences?\" \"Here we are.\" \"What are you doing?\" \"It's cold in here.\" \"It's fine.\" \"No, it's not.\" \"What's going on here?\" \"Oh, that's Cat's new boyfriend.\" \"Boyfriend?\" \"She just broke up with me.\" \"Um, a month ago.\" \"Hey, she just tried kissing me two weeks ago.\" \"Well, she'll date anything.\" \"No offense to me.\" \"None taken.\" \"Remember when she dated\" \"Josh Brolin?\" \"Remember when she dated Josh Brolin's character in\" \"\"Goonies\"?\" \"Remember when she dated that sculpture of our hospital's founder, mr.\" \"Childrens?\" \"In \"Goonies\"?\" \"No, he wasn't in \"Goonies\".\" \"He wasn't in \"Goonies\"?\" \"Who was it?\" \"Who \" \"Oh, you know who I'm thinking of?\" \"Huh?\" \"Josh Brolin.\" \"She'll date anything.\" \"What does that make us?\" \"The two hottest doctors in here.\" \"You are so true.\" \"I feel dizzy.\" \"I want to get deep inside of you.\" \"Congratulations,\" \"Dr. Von Sdow.\" \"You are the first doctor from\" \"Childrens ever to receive the\" \"Nobel prize.\" \"Well, it was just for medicine.\" \"They offered me the peace prize, but I turned it down 'cause I don't want the bitches to think\" \"I'm a gaylord.\" \"Well, Max, I'm very busy, so...\" \"Right.\" \"I'm Audi RS4 with the gold package.\" \"M. S., right?\" \"Excuse me?\" \"You have M. S.?\" \"A touch.\" \"Probably a little Parkinson's in there, too, right?\" \"Little bit.\" \"How much\" \"Cerebral palsy do you have?\" \"I have a splash.\" \"It's going around.\" \"Why?\" \"You sitting down?\" \"I can cure you.\" \"Leave the pee.\" \"Okay.\" \"Careful.\" \"You know I can't fix your handicapped emotions, right?\" \"Previously on\" \"\"childrens hospital\"...\" \"And so it is my great pleasure and honor to open up\" \"Childrens hospital's very own genital ward.\" \"Okay, I-I'm sorry.\" \"I just got to get my head\" \"Wrapped around this.\" \"So, you're saying that there is a law that says we cannot test new drugs on rare pandas?\" \"That's what I am saying.\" \"Really?\" \"Mm-hmm.\" \"Aah!\" \"W- while we're on the subject, can I ask you another question?\" \"Mm-hmm.\" \"Is there a law against jerking off in a government-office cubicle?\" \"This genital ward will put\" \"Childrens hospital on the forefront of modern... genital technology and genital research.\" \"You sitting down?\" \"I can cure you.\" \"You know that visiting doctor, the neuro-specialist?\" \"Dr. Max von sydow?\" \"Yeah, I heard he won a\" \"Nobel peace prize.\" \"Funny - he doesn't look like a fag.\" \"He says he can cure me.\" \"What?\" \"!\" \"Hold on, chief.\" \"If he does that, you might lose your sexy strut.\" \"Strut.\" \"Maybe, but I have no feeling in my pelvis.\" \"I'll feel that s**t for you.\" \"You're sweet.\" \"But I want to be normal, owen.\" \"I\" \" I want to be like everyone else.\" \"That's a kick in the nuts to all those young girls out there that were never called \"special\" or\" \"\"crippled\" or \"handi-crippled. \"\" \"Bet you say that to all the girls.\" \"N- no.\" \"I don't.\" \"Oh, yeah.\" \"I guess you wouldn't.\" \"I don't know other handicapped people.\" \"Right.\" \"Chief, you have a gift.\" \"Please, don't ruin it.\" \"Now, how about that hand job?\" \"Dora?\" \"Hi.\" \"Sorry to wake you.\" \"I just want to get some alone time while your mom is at her\" \"AA meeting.\" \"Can you look under the bed for Dracular?\" \"Oh, yeah.\" \"I'm gonna look under everything.\" \"I got to go peep.\" \"Can't you wait five minutes?\" \"I can count to 100.\" \"Okay.\" \"One...\" \"Two...\" \"Three...\" \"Oh, yeah.\" \"Cup the balls.\" \"Hey!\" \"I'm challenging you to a\" \"\"healing power of laughter\"- off.\" \"You have one week to perform a successful operation using only the healing power of laughter.\" \"If you fail, you hang up your balloony pants forever.\" \"I won't fail.\" \"Ooh, I won't.\" \"Fine.\" \"If I fail...\" \"I leave medicine altogether.\" \"No.\" \"You go f**k yourself.\" \"Attention, staff.\" \"I am speaking on the P. A.\" \"system.\" \"That's what I'm doing right now in a clear and authoritative voice.\" \"That's it.\" \"This kid's getting a vasectomy.\" \"No.\" \"Don't.\" \"Diego?\" \"My mom says that Israel's an outdated concept.\" \"Thank you for watching these two episodes of\" \"\"Childrens Hospital. \"\" \"A lot of people ask me where\" \"I got the idea for\" \"\"Childrens Hospital\", and it's a funny story, actually.\" \"Uh, my daughter - my 2-year-old daughter dislocated her shoulder, and we had to bring\" \"Her to a children's hospital.\" \"And I just remember, like, the waiting room was terrifying.\" \"It was, uh, full of nervous parents, most of them crying,\" \"You know, and, uh...\" \"All pale.\" \"This isn't the funny part.\" \"But, um, this gurney - this tiny little gurney comes crashing through the door, and it's surrounded by doctors and nurses, and they're yelling, and they look really scared, and there's this...\" \"Tiny body on the gurney.\" \"The funny part's coming.\" \"And, uh, and there's this life flight pilot, this helicopter pilot behind the gurney.\" \"And, uh, and he looks at me, and he goes, \"Mondays, right?\"\" \"He didn't actually say that.\" \"So, then I made a tv show about it.\"\n",
"remark": {
"pile_set_name": "OpenSubtitles"
},
"sub_path": "opensubtitles/train"
}
```
## 引文
```
@misc{conghui2022opendatalab,
title={OpenDataLab: Empowering General Artificial Intelligence with Open Datasets},
author={Conghui He, Wei Li, Zhenjiang Jin, Bin Wang, Chao Xu, Dahua Lin},
journal={https://opendatalab.com/},
year={2022}
}
```
## Download dataset
:modelscope-code[]{type="git"}

显示名称：Pile-OpenSubtitles
许可证：
- MIT协议
任务类型：
- 自然语言生成
- 语言建模
媒体类型：
- 文本
标签类型：
- 英语语料库
标签：无
发布方：
- EleutherAI
发布日期：2023年7月18日
发布网址：https://pile.eleuther.ai/
论文网址：无

---
# 数据集介绍
## 数据集简介
Pile-OpenSubtitles数据集隶属于The Pile项目，是专为大语言模型（Large Language Model）打造的训练数据集。本数据集依托OpenSubtitles项目的字幕文本资源构建。
OpenSubtitles作为开源字幕资源库，收录了海量影视剧集的字幕文件。Pile-OpenSubtitles数据集通过复用这些字幕文本，为大语言模型提供了丰富的对话类及会话式文本训练资源。
本数据集囊括多元类型的对话内容，覆盖各异的主题与语言风格，包含影视剧中的各类对白，涉及日常交际、科技探讨等广泛领域的文本。
借助本数据集，研究人员与开发者可训练大语言模型理解并生成会话式文本，为对话生成、情感分析、对话系统等领域的应用提供支撑。
## 数据集内容
### 数据规模说明
Pile-OpenSubtitles数据集总数据量为6.7吉字节（GB）。
### 数据样例

{
"id": "146452926",
"source_id": "",
"doc_id": "27239921",
"data_type": "text",
"data_source": "pile",
"data_url": "enwiki-c4-pile-ccnews",
"content": ""We're breaking up." "It is not working out between us." " We're breaking up." " Not here." "Not in front of the kids." " We're breaking up." " I can tell." "I'm a doctor." " You know we're breaking up, right?" " Oh, yeah." " I have to break up with both of you." " What?" " You know we're breaking up, right?" " Oh, yeah." "It's not working for me." "It's over." "Again?" "It's over." "Again?" "Okay, don't be so dramatic." "We're breaking up." "We're break" "Okay." "I'm sorry." "I've forgotten your name." "We're breaking up!" "I can't hear you!" "No, I know that you can hear me 'cause I can hear you." "We're breaking up." "Okay." "It's over." "Will you e-mail me because I might forget." "Things haven't been working out, and I just " "I'm breaking up with you." "Beat you to it." "Boom!" "We're breaking up." "Huh?" "Can't it wait 20 seconds?" "10 seconds." "No." "I got to go." "Aah!" "Wait!" "Attention, staff." "There's a gigantic snake moving below the elevators." "Love." "Why is it that when you're in" "It, you can't remember not" "Being in it?" "And when you're not in it, it's" "Like, "what is it?"" "Why is it not just in and of" "Itself, in or out?" "I should do my own podcast." "This looks like" "Esperanza the maid." "well, it's like a big maid" "And a little maid." "I don't like chocolate people." "Oh, don't say that in public, honey." "Hey." "Dr. Black, you spend a lot of time treating Nicky, but his advanced-aging disease seems to be getting worse." "I have gout." "Why don't you go on down to the cafeteria and get us an eight-pack of smirnoff ice, yeah?" "Can you get on with the exam, please, before Nicky gets any older?" "God, look at those eyes." "I have heat vision!" "God, I just wish I knew you when you were young." "He's 6." "Well, then I wish I knew you when you were 4." "Here's a question." "What?" "Are space bugs real?" "Do you like new experiences?" "Here we are." "What are you doing?" "It's cold in here." "It's fine." "No, it's not." "What's going on here?" "Oh, that's Cat's new boyfriend." "Boyfriend?" "She just broke up with me." "Um, a month ago." "Hey, she just tried kissing me two weeks ago." "Well, she'll date anything." "No offense to me." "None taken." "Remember when she dated" "Josh Brolin?" "Remember when she dated Josh Brolin's character in" ""Goonies"?" "Remember when she dated that sculpture of our hospital's founder, mr." "Childrens?" "In "Goonies"?" "No, he wasn't in "Goonies"." "He wasn't in "Goonies"?" "Who was it?" "Who " "Oh, you know who I'm thinking of?" "Huh?" "Josh Brolin." "She'll date anything." "What does that make us?" "The two hottest doctors in here." "You are so true." "I feel dizzy." "I want to get deep inside of you." "Congratulations," "Dr. Von Sdow." "You are the first doctor from" "Childrens ever to receive the" "Nobel prize." "Well, it was just for medicine." "They offered me the peace prize, but I turned it down 'cause I don't want the bitches to think" "I'm a gaylord." "Well, Max, I'm very busy, so..." "Right." "I'm Audi RS4 with the gold package." "M. S., right?" "Excuse me?" "You have M. S.?" "A touch." "Probably a little Parkinson's in there, too, right?" "Little bit." "How much" "Cerebral palsy do you have?" "I have a splash." "It's going around." "Why?" "You sitting down?" "I can cure you." "Leave the pee." "Okay." "Careful." "You know I can't fix your handicapped emotions, right?" "Previously on" ""childrens hospital"..." "And so it is my great pleasure and honor to open up" "Childrens hospital's very own genital ward." "Okay, I-I'm sorry." "I just got to get my head" "Wrapped around this." "So, you're saying that there is a law that says we cannot test new drugs on rare pandas?" "That's what I am saying." "Really?" "Mm-hmm." "Aah!" "W- while we're on the subject, can I ask you another question?" "Mm-hmm." "Is there a law against jerking off in a government-office cubicle?" "This genital ward will put" "Childrens hospital on the forefront of modern... genital technology and genital research." "You sitting down?" "I can cure you." "You know that visiting doctor, the neuro-specialist?" "Dr. Max von sydow?" "Yeah, I heard he won a" "Nobel peace prize." "Funny - he doesn't look like a fag." "He says he can cure me." "What?" "!" "Hold on, chief." "If he does that, you might lose your sexy strut." "Strut." "Maybe, but I have no feeling in my pelvis." "I'll feel that s**t for you." "You're sweet." "But I want to be normal, owen." "I" " I want to be like everyone else." "That's a kick in the nuts to all those young girls out there that were never called "special" or" ""crippled" or "handi-crippled. "" "Bet you say that to all the girls." "N- no." "I don't." "Oh, yeah." "I guess you wouldn't." "I don't know other handicapped people." "Right." "Chief, you have a gift." "Please, don't ruin it." "Now, how about that hand job?" "Dora?" "Hi." "Sorry to wake you." "I just want to get some alone time while your mom is at her" "AA meeting." "Can you look under the bed for Dracular?" "Oh, yeah." "I'm gonna look under everything." "I got to go peep." "Can't you wait five minutes?" "I can count to 100." "Okay." "One..." "Two..." "Three..." "Oh, yeah." "Cup the balls." "Hey!" "I'm challenging you to a" ""healing power of laughter"- off." "You have one week to perform a successful operation using only the healing power of laughter." "If you fail, you hang up your balloony pants forever." "I won't fail." "Ooh, I won't." "Fine." "If I fail..." "I leave medicine altogether." "No." "You go f**k yourself." "Attention, staff." "I am speaking on the P. A." "system." "That's what I'm doing right now in a clear and authoritative voice." "That's it." "This kid's getting a vasectomy." "No." "Don't." "Diego?" "My mom says that Israel's an outdated concept." "Thank you for watching these two episodes of" ""Childrens Hospital. "" "A lot of people ask me where" "I got the idea for" ""Childrens Hospital", and it's a funny story, actually." "Uh, my daughter - my 2-year-old daughter dislocated her shoulder, and we had to bring" "Her to a children's hospital." "And I just remember, like, the waiting room was terrifying." "It was, uh, full of nervous parents, most of them crying," "You know, and, uh..." "All pale." "This isn't the funny part." "But, um, this gurney - this tiny little gurney comes crashing through the door, and it's surrounded by doctors and nurses, and they're yelling, and they look really scared, and there's this..." "Tiny body on the gurney." "The funny part's coming." "And, uh, and there's this life flight pilot, this helicopter pilot behind the gurney." "And, uh, and he looks at me, and he goes, "Mondays, right?"" "He didn't actually say that." "So, then I made a tv show about it."
",
"remark": {
"pile_set_name": "OpenSubtitles"
},
"sub_path": "opensubtitles/train"
}

## 引用文献

@misc{conghui2022opendatalab,
title={OpenDataLab: Empowering General Artificial Intelligence with Open Datasets},
author={Conghui He, Wei Li, Zhenjiang Jin, Bin Wang, Chao Xu, Dahua Lin},
journal={https://opendatalab.com/},
year={2022}
}

## 数据集下载
:modelscope-code[]{type="git"}