zhisheng01/SpatialSoundQA

# SpatialSoundQA

## Binaural Audio Dataset
We provide Audioset as anechoic (monaural audio) sound sources, and we apply reverberation to them to generate binaural audio.

We use the Audioset balanced set as training data, and the eval set as evaluation data.
For reverberation, please download [binaural.zip](https://huggingface.co/datasets/zhisheng01/SpatialSoundQA/blob/main/binaural.zip)

### How to apply convolution to create binaural audio from a monaural recording?
```python
import numpy as np
import soundfile as sf
from scipy import signal

audio, sr = sf.read('demo.wav')
reverb = np.load('demo_revber.npy')
binaural = signal.fftconvolve(audio, reverb, mode='full')

# or you can use torchaudio to achieve this:
# audio = torch.from_numpy(audio)
# reverb = torch.from_numpy(reverb)
# binaural = torchaudio.functional.fftconvolve(audio, reverb, mode='full')
```

## QA dataset
According to the [paper](https://arxiv.org/abs/2402.01591), we provide Question-Answering pairs for stage [1](https://huggingface.co/datasets/zhisheng01/SpatialSoundQA/blob/main/train-qa-stage1.json), [2](https://huggingface.co/datasets/zhisheng01/SpatialSoundQA/blob/main/train-qa-stage2.json), and [3](https://huggingface.co/datasets/zhisheng01/SpatialSoundQA/blob/main/train-qa-stage3.json).

## TODO
1. Provide Audioset balanced set and evaluation set.
2. Upload QA eval sets.

---
license: cc-by-nc-4.0
---

原始信息汇总

SpatialSoundQA 数据集概述

数据集组成

• 音频数据: 使用 Audioset 的无回声（单声道音频）作为声源，并通过混响处理生成双耳音频。

  • 训练数据: 使用 Audioset 平衡集。
  • 评估数据: 使用 Audioset 评估集。
  • 混响文件: 可通过下载 binaural.zip 获取。

• 问答数据: 提供三个阶段的问答对。

  • 阶段1: train-qa-stage1.json
  • 阶段2: train-qa-stage2.json
  • 阶段3: train-qa-stage3.json

数据处理方法

• 混响生成: 使用 numpy 和 soundfile 库，通过 signal.fftconvolve 函数将单声道音频与混响数据进行卷积以生成双耳音频。
• 可选方法: 使用 torchaudio 库进行相同的卷积操作。

待办事项

• 提供 Audioset 平衡集和评估集。
• 上传问答评估集。

许可证

• 本数据集遵循 CC-BY-NC-4.0 许可证。