Babillage

# Babillage

Babillage is a multimodal benchmark dataset introduced along with **MoshiVis** ([Project Page](https://kyutai.org/moshivis) | [arXiv](https://arxiv.org/abs/2503.15633)), containing three common vision-language benchmarks converted in spoken form, for the evaluation of Vision Speech Models.
For each benchmark (COCO-Captions, OCR-VQA, VQAv2), we first reformat the text question-answer pairs into a more conversational dialogue, and then convert them using a text-to-speech pipeline, using a
consistent synthetic voice for the answer (assistant) and diverse voices for the question (user).
We provide the resulting spoken samples in the dataset.
The dataset is designed for research in conversational AI, spoken VQA, and multimodal dialogue systems.

## Dataset Summary

Babillage contains the following dataset config and splits:

- **Conversational COCO (CoCOCO) Captions** Image captions from COCO synthesized into speech. Based on [COCO](https://cocodataset.org/#home).
- **validation** split (`25010` samples, `5000` unique images) following Karpathy splits (including spoken answers)
- **test** split (`25010` samples, `5000` unique images) following Karpathy splits (including spoken answers)
- **Conversational OCR-VQA (CoOCR-VQA)**: Originally designed for text-based VQA involving optical character recognition,
Babillage reformats the questions and answers into a more conversational style and generates speech for them. Based on [OCR-VQA](https://ocr-vqa.github.io/).
- **validation** split (`100032` samples, `20730` unique images) (including spoken answers)
- **test** split (`100424` samples, `20795` unique images) (including spoken answers)
- **Conversational VQAv2 (CoVQAv2)**: A well-known VQA dataset, reformatted for dialogue-style interaction and
converted into speech with different speakers. Based on [VQAv2](https://visualqa.org/).
- **validation** split (`214354` samples, `40504` unique images)

## Data Format

Each dataset sample consists of:
- **Indexing:** A way to refer to the corresponding sample in the respective base dataset (OCR-VQA, VQAv2, or COCO Captions).
- `sample_id`: Index of the corresponding sample in the base dataset
- `image_id`: Image ID (only for CoOCR-VQA and CoCOCO)
- **Question in audio form**:
- `question_audio`: The question in audio form.
- `question_transcript`: A list of words constituting the text transcript of `question_audio`
- `question_alignment`: A list of time spans tuples `(start, end)` consituting the alignment of each word in `question_transcript` to the audio
- **Answer in audio form:** (only for CoOCR-VQA and CoCOCO)
- `answer_audio`: The answer in audio form
- `answer_transcript`: A list of words constituting the text transcript of `answer_audio`
- `answer_alignment`: A list of time spans tuple `(start, end)` consituting the alignment of each word in `answer_transcript` to the audio

## Loading the dataset
In the following code snippets, we showcase how to merge Babillage splits with their corresponding
original vision-language dataset, to map the audio samples with the corresponding image and ground-truth.
Note that all audio files in this dataset are stored as `ogg`. If you want to obtain them as uncompressed
`wav`, you can use the following snippet:

```python
import io
import soundfile as sf
import marimo as mo # example display with marimo notebook

def audio_widget(decoded_audio) -> Any:
"""A small snippet to convert to uncompressed audio (wav) and display the sample with marimo"""
byte_io = io.BytesIO(bytes())
sf.write(byte_io, decoded_audio["array"], samplerate=decoded_audio["sampling_rate"], format="wav")
return mo.audio(byte_io)
```

Below you'll also find a convenient code snippet to display one data sample from the Babillage dataset
```python

from random import randint
import numpy as np

def display_one(dataset):
ditem = dataset[randint(0, len(dataset) - 1)]
print("Ground-truth:", ditem["gt"])
print("Question alignment:", [(w, ts) for w, ts in zip(ditem["question_transcript"],
ditem["question_alignment"])])
if "answer_audio" in ditem:
print("Answer alignment:", [(w, ts) for w, ts in zip(ditem["answer_transcript"],
ditem["answer_alignment"])])
return (mo.image(np.array(ditem["image"])),
audio_widget(ditem["question_audio"]),
audio_widget(ditem["answer_audio"]))
else:
return (mo.image(np.array(ditem["image"])),
audio_widget(ditem["question_audio"]))
```

#### CoCOCO

```python
import datasets
split = "validation"
coCOCO = datasets.load_dataset("kyutai/Babillage", "coco", split=split)

# Merge with COCO original information
original_columns = (
datasets.load_dataset("HuggingFaceM4/COCO", "2014", split=split)
.flatten()
.select_columns(["sentences.raw", "image", "cocoid"])
.cast_column('image', datasets.Image(decode=False))
)
coCOCO = coCOCO.add_column("og_image_id", original_columns["cocoid"])
coCOCO = coCOCO.add_column("gt", original_columns["sentences.raw"])
coCOCO = coCOCO.add_column("image", original_columns["image"])
coCOCO = coCOCO.cast_column('image', datasets.Image(decode=True))
del original_columns

# Display one sample
display_one(coCOCO)
```

#### CoOCR-VQA

```python
import datasets
split = "validation"
coOCRVQA = datasets.load_dataset("kyutai/Babillage", "ocrvqa", split=split)
original_columns = (
datasets.load_dataset("howard-hou/OCR-VQA", split=split)
.flatten()
.select_columns(["image_id", "questions", "answers", "image"])
.cast_column('image', datasets.Image(decode=False))
)

num_samples_per_image = [len(x) for x in original_columns["questions"]]
coOCRVQA = coOCRVQA.add_column("og_image_id", [
id for n, id in zip(num_samples_per_image, original_columns["image_id"])
for _ in range(n)])
coOCRVQA = coOCRVQA.add_column(
"gt", [[quest, ans]
for qlst, alst in zip(original_columns["questions"], original_columns["answers"])
for quest, ans in zip(qlst, alst)])
coOCRVQA = coOCRVQA.add_column("image", [
img for n, img in zip(num_samples_per_image, original_columns["image"])
for _ in range(n)])
coOCRVQA = coOCRVQA.cast_column('image', datasets.Image(decode=True))

# Display one sample
display_one(coOCRVQA)
```

#### CoVQAv2
```python
import datasets
split = "validation"
coVQA = datasets.load_dataset("kyutai/Babillage", "vqav2", split=split)
original_columns = (
datasets.load_dataset("lmms-lab/VQAv2", split=split)
.flatten()
.select_columns(["image_id", "question", "answers", "image"])
.cast_column('image', datasets.Image(decode=False)))

coVQA = coVQA.add_column("og_image_id", original_columns["image_id"])
coVQA = coVQA.add_column("gt", original_columns["answers"])
coVQA = coVQA.add_column("gt_question", original_columns["question"])
coVQA = coVQA.add_column("image", original_columns["image"])
coVQA = coVQA.cast_column('image', datasets.Image(decode=True))

# Display one sample
display_one(coVQA)
```

## Licensing

Babillage is licensed under the **CC-BY 4.0** license.

## Citation

If you use this dataset, please cite:

```
@article{kyutai2025moshivis,
author = {Amélie Royer and Moritz Böhle and Gabriel de Marmiesse and
Laurent Mazaré and Alexandre Défossez and Neil Zeghidour and Patrick Pérez},
year = {2025},
title = {Vision-Speech Models: Teaching Speech Models to Converse about Images},
journal = {ArXiv},
url = {https://arxiv.org/abs/2503.15633}
}

@misc{babillage2025,
title={{Babillage: A Conversational and Spoken VQA Dataset}},
author = {Amélie Royer, Moritz Böhle, Gabriel de Marmiesse, Laurent Mazaré, Alexandre Défossez, Neil Zeghidour, Patrick Pérez},
year={2025},
url={https://huggingface.co/datasets/kyutai/babillage}
}
```

# Babillage数据集
Babillage是伴随**MoshiVis**（[项目页面](https://kyutai.org/moshivis) | [arXiv论文](https://arxiv.org/abs/2503.15633)）推出的多模态基准数据集，包含三个经口语化转换的常见视觉语言基准任务，用于评估视觉语音模型（Vision Speech Models）。针对每个基准任务（COCO-Captions、OCR-VQA、VQAv2），我们首先将文本问答对重构为更自然的对话式交互形式，随后通过文本转语音（text-to-speech, TTS）流水线生成对应语音，其中回答（助手角色）采用统一的合成语音，而提问（用户角色）则使用多样化的语音。本数据集提供了生成后的语音样本，其设计初衷是支持对话式人工智能、口语化视觉问答（spoken VQA）以及多模态对话系统领域的研究。

## 数据集概览
Babillage包含如下数据集配置与子集划分：
- **对话式COCO（CoCOCO）图像描述**：将COCO的图像描述语音合成得到。基于[COCO](https://cocodataset.org/#home)数据集。
- **验证集（validation）**：共25010条样本，涵盖5000张独特图像，遵循Karpathy划分规则（包含语音化回答）
- **测试集（test）**：共25010条样本，涵盖5000张独特图像，遵循Karpathy划分规则（包含语音化回答）
- **对话式OCR-VQA（CoOCR-VQA）**：该数据集最初面向涉及光学字符识别（Optical Character Recognition, OCR）的文本型视觉问答任务，Babillage将其中的问答对重构为对话式风格并生成对应语音。基于[OCR-VQA](https://ocr-vqa.github.io/)数据集。
- **验证集**：共100032条样本，涵盖20730张独特图像（包含语音化回答）
- **测试集**：共100424条样本，涵盖20795张独特图像（包含语音化回答）
- **对话式VQAv2（CoVQAv2）**：经典的视觉问答基准数据集，经重构适配对话式交互形式并转换为多发言人语音。基于[VQAv2](https://visualqa.org/)数据集。
- **验证集**：共214354条样本，涵盖40504张独特图像

## 数据格式
每条数据集样本包含以下内容：
- **索引信息**：用于指向对应基础数据集（OCR-VQA、VQAv2或COCO图像描述）中的样本
- `sample_id`：基础数据集中对应样本的索引
- `image_id`：图像ID（仅适用于CoOCR-VQA和CoCOCO子集）
- **语音形式的提问**：
- `question_audio`：提问的音频文件
- `question_transcript`：组成提问音频文本转录结果的单词列表
- `question_alignment`：由时间跨度元组`(start, end)`组成的列表，用于将`question_transcript`中的每个单词与音频片段对齐
- **语音形式的回答**：（仅适用于CoOCR-VQA和CoCOCO子集）
- `answer_audio`：回答的音频文件
- `answer_transcript`：组成回答音频文本转录结果的单词列表
- `answer_alignment`：由时间跨度元组`(start, end)`组成的列表，用于将`answer_transcript`中的每个单词与音频片段对齐

## 数据集加载
以下代码片段展示了如何将Babillage的子集与对应的原始视觉语言数据集进行合并，以实现音频样本与对应图像及真实标注的映射。请注意，本数据集内所有音频文件均以`ogg`格式存储，若需转换为未压缩的`wav`格式，可使用如下代码片段：
python
import io
import soundfile as sf
import marimo as mo # 使用marimo笔记本的示例展示代码

def audio_widget(decoded_audio) -> Any:
"""将音频转换为未压缩格式（wav）并使用marimo展示样本的小工具"""
byte_io = io.BytesIO(bytes())
sf.write(byte_io, decoded_audio["array"], samplerate=decoded_audio["sampling_rate"], format="wav")
return mo.audio(byte_io)

下文还提供了一段便捷的代码片段，用于可视化Babillage数据集中的单条样本：
python
from random import randint
import numpy as np

def display_one(dataset):
ditem = dataset[randint(0, len(dataset) - 1)]
print("真实标注:", ditem["gt"])
print("提问对齐信息:", [(w, ts) for w, ts in zip(ditem["question_transcript"], ditem["question_alignment"])])
if "answer_audio" in ditem:
print("回答对齐信息:", [(w, ts) for w, ts in zip(ditem["answer_transcript"], ditem["answer_alignment"])])
return (mo.image(np.array(ditem["image"])), audio_widget(ditem["question_audio"]), audio_widget(ditem["answer_audio"]))
else:
return (mo.image(np.array(ditem["image"])), audio_widget(ditem["question_audio"]))

#### CoCOCO子集
python
import datasets
split = "validation"
coCOCO = datasets.load_dataset("kyutai/Babillage", "coco", split=split)

# 与原始COCO数据集合并信息
original_columns = (
datasets.load_dataset("HuggingFaceM4/COCO", "2014", split=split)
.flatten()
.select_columns(["sentences.raw", "image", "cocoid"])
.cast_column('image', datasets.Image(decode=False))
)
coCOCO = coCOCO.add_column("og_image_id", original_columns["cocoid"])
coCOCO = coCOCO.add_column("gt", original_columns["sentences.raw"])
coCOCO = coCOCO.add_column("image", original_columns["image"])
coCOCO = coCOCO.cast_column('image', datasets.Image(decode=True))
del original_columns

# 展示单条样本
display_one(coCOCO)

#### CoOCR-VQA子集
python
import datasets
split = "validation"
coOCRVQA = datasets.load_dataset("kyutai/Babillage", "ocrvqa", split=split)
original_columns = (
datasets.load_dataset("howard-hou/OCR-VQA", split=split)
.flatten()
.select_columns(["image_id", "questions", "answers", "image"])
.cast_column('image', datasets.Image(decode=False))
)

num_samples_per_image = [len(x) for x in original_columns["questions"]]
coOCRVQA = coOCRVQA.add_column("og_image_id", [
id for n, id in zip(num_samples_per_image, original_columns["image_id"])
for _ in range(n)])
coOCRVQA = coOCRVQA.add_column(
"gt", [[quest, ans]
for qlst, alst in zip(original_columns["questions"], original_columns["answers"])
for quest, ans in zip(qlst, alst)])
coOCRVQA = coOCRVQA.add_column("image", [
img for n, img in zip(num_samples_per_image, original_columns["image"])
for _ in range(n)])
coOCRVQA = coOCRVQA.cast_column('image', datasets.Image(decode=True))

# 展示单条样本
display_one(coOCRVQA)

#### CoVQAv2子集
python
import datasets
split = "validation"
coVQA = datasets.load_dataset("kyutai/Babillage", "vqav2", split=split)
original_columns = (
datasets.load_dataset("lmms-lab/VQAv2", split=split)
.flatten()
.select_columns(["image_id", "question", "answers", "image"])
.cast_column('image', datasets.Image(decode=False)))

coVQA = coVQA.add_column("og_image_id", original_columns["image_id"])
coVQA = coVQA.add_column("gt", original_columns["answers"])
coVQA = coVQA.add_column("gt_question", original_columns["question"])
coVQA = coVQA.add_column("image", original_columns["image"])
coVQA = coVQA.cast_column('image', datasets.Image(decode=True))

# 展示单条样本
display_one(coVQA)

## 授权协议
Babillage采用**CC-BY 4.0**协议发布。

## 引用声明
若您使用本数据集，请引用如下文献：

@article{kyutai2025moshivis,
author = {Amélie Royer and Moritz Böhle and Gabriel de Marmiesse and
Laurent Mazaré and Alexandre Défossez and Neil Zeghidour and Patrick Pérez},
year = {2025},
title = {Vision-Speech Models: Teaching Speech Models to Converse about Images},
journal = {ArXiv},
url = {https://arxiv.org/abs/2503.15633}
}

@misc{babillage2025,
title={{Babillage: A Conversational and Spoken VQA Dataset}},
author = {Amélie Royer, Moritz Böhle, Gabriel de Marmiesse, Laurent Mazaré, Alexandre Défossez, Neil Zeghidour, Patrick Pérez},
year={2025},
url={https://huggingface.co/datasets/kyutai/babillage}
}