arkimjh/social-benchmark

---
license: cc-by-4.0
task_categories:
- video-text-to-text
- visual-question-answering
language:
- en
tags:
- video
- social-interaction
- benchmark
- multi-person
- gaze
- gesture-recognition
size_categories:
- 1K<n<10K
---

# Social Benchmark Dataset

## Description

Social Benchmark is a video-based social understanding benchmark dataset containing **6,433 video clips** from 3 evaluation sets: MMSI, Online-MMSI, and original TVQA+. Each benchmark tests different aspects of social reasoning in multi-person video scenarios.

## Dataset Statistics

| Benchmark | Videos | Size | JSON Files |
|-----------|--------|------|------------|
| MMSI | 1,708 | ~377 MB | 3 (mpp, pcr, sti) |
| Online-MMSI | 1,708 | ~2.9 GB | 3 (mpp, pcr, sti) |
| original TVQA+ | 3,017 | ~540 MB | 1 (eval) |
| **Total** | **6,433** | **~3.8 GB** | **7** |

## Benchmark Tasks

- **MPP** (Multi-Person Prediction): Predict social behaviors of multiple persons
- **PCR** (Person-Centric Reasoning): Reason about individual person's social role
- **STI** (Social Temporal Inference): Infer social dynamics over time
- **TVQA+**: Video question answering with spatial-temporal grounding

## Dataset Structure

```
social-benchmark/
├── json/
│ ├── mmsi_test_mpp_original.json
│ ├── mmsi_test_pcr_original.json
│ ├── mmsi_test_sti_original.json
│ ├── online_mmsi_test_mpp_extended.json
│ ├── online_mmsi_test_pcr_extended.json
│ ├── online_mmsi_test_sti_extended.json
│ └── tvqa_plus_eval.json
└── tars/
├── mmsi_videos_part001.tar
├── online_mmsi_videos_part001.tar
└── original_tvqa_plus_videos_part001.tar
```

## Usage

```python
from huggingface_hub import snapshot_download

# Download everything
snapshot_download(
repo_id="arkimjh/social-benchmark",
repo_type="dataset",
local_dir="./social-benchmark"
)

# Extract video archives
import tarfile
from pathlib import Path

for tar_file in Path("./social-benchmark/tars").glob("*.tar"):
with tarfile.open(tar_file) as tf:
tf.extractall("./social-benchmark/videos/")
```

## Data Sources

- **MMSI**: Multi-Modal Social Interaction benchmark (offline setting, original videos)
- **Online-MMSI**: Multi-Modal Social Interaction benchmark (online setting, original videos)
- **original TVQA+**: TV show QA with compositional spatio-temporal grounding (original videos)

## Citation

TBD