Music-Instruct

# Music Instruct (MI) Dataset

This is the dataset used to train and evaluate the MusiLingo model.
This dataset contains Q&A pairs related
to individual musical compositions, specifically
tailored for open-ended music queries. It originates
from the music-caption pairs in the MusicCaps
dataset.
The MI dataset was created through prompt engineering and applying few-shot learning techniques
to GPT-4. More details on dataset generation can be found in our paper *[MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response
](https://arxiv.org/abs/2309.08730)*.

The resulting MI dataset consists of two versions:
v1 (short questions), with 27,540 Q&A pairs seeking comprehensive
details about musical snippets including but not limited to emotion, instrument, vocal track, tempo, and genre etc., often yielding concise one or two-sentence responses. In contrast, v2
comprises 32,953 Q&A pairs featuring more general questions about the musical pieces (long questions), resulting
in typically more extensive responses that serve as
paraphrased renditions of the original caption.

## Evaluation and dataset SPlittion
You can use all (or the long/short partition of) the Q\&A pairs of which audio is in the training split of AudioSet as MI training set and use the short QA and long QA with audio in evaluation split of AudioSet as two testingsets separately.

```
# training set
ds_mixed_train = MIDataset(processor, '/content/drive/MyDrive/music_data', split='train', question_type='all')
ds_long_train = MIDataset(processor, '/content/drive/MyDrive/music_data', split='train', question_type='long')
ds_short_train = MIDataset(processor, '/content/drive/MyDrive/music_data', split='train', question_type='short')
# testing set for short QA
ds_short = MIDataset(processor, '/content/drive/MyDrive/music_data', split='test', question_type='short')
# testing set for long QA
ds_long = MIDataset(processor, '/content/drive/MyDrive/music_data', split='test', question_type='long')
```

And the evaluation includes BLEU, METEOR, ROUGE, and Bert-Score.

## Citation
```
@article{deng2023musilingo,
title={MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response},
author={Deng, Zihao and Ma, Yinghao and Liu, Yudong and Guo, Rongchen and Zhang, Ge and Chen, Wenhu and Huang, Wenhao and Benetos, Emmanouil},
journal={arXiv preprint arXiv:2309.08730},
year={2023}
}
```

# 音乐指令（Music Instruct，MI）数据集

本数据集用于训练与评估MusiLingo模型。数据集包含针对单首音乐作品的问答对，专门适配开放式音乐查询场景，其数据源头来自MusicCaps数据集中的音乐-标题配对样本。

MI数据集通过提示工程（prompt engineering）与少样本学习（few-shot learning）技术结合GPT-4构建而成。有关数据集生成的详细细节可参阅我们的论文《MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response》（arxiv预印本编号：2309.08730，链接：https://arxiv.org/abs/2309.08730）。

最终生成的MI数据集包含两个版本：v1为短问题版本，包含27540组问答对，用于查询音乐片段的多维度细节，涵盖情感、乐器、人声轨道、节拍速度、曲风等范畴，回答通常为简洁的1至2句话。与之相对，v2为长问题版本，包含32953组问答对，问题更具普适性，回答篇幅更长，本质是对原始标题的释义性重述。

## 评估与数据集划分

可将AudioSet训练划分中包含音频的全部（或长、短分区）问答对作为MI训练集；将AudioSet评估划分中包含音频的短问答与长问答分别作为两个独立测试集。

# training set
ds_mixed_train = MIDataset(processor, '/content/drive/MyDrive/music_data', split='train', question_type='all')
ds_long_train = MIDataset(processor, '/content/drive/MyDrive/music_data', split='train', question_type='long')
ds_short_train = MIDataset(processor, '/content/drive/MyDrive/music_data', split='train', question_type='short')
# testing set for short QA
ds_short = MIDataset(processor, '/content/drive/MyDrive/music_data', split='test', question_type='short')
# testing set for long QA
ds_long = MIDataset(processor, '/content/drive/MyDrive/music_data', split='test', question_type='long')

评估指标包含BLEU、METEOR、ROUGE以及Bert-Score。

## 引用

@article{deng2023musilingo,
title={MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response},
author={Deng, Zihao and Ma, Yinghao and Liu, Yudong and Guo, Rongchen and Zhang, Ge and Chen, Wenhu and Huang, Wenhao and Benetos, Emmanouil},
journal={arXiv preprint arXiv:2309.08730},
year={2023}
}