DDDDeno/PSVBench

---
license: cc-by-nc-4.0
task_categories:
- video-classification
- question-answering
task_ids:
- multiple-choice-qa
configs:
- config_name: default
data_files:
- split: eval
path: "qa/eval.json"
---
# PSVBench

This is an initial version for reviewing.

## Dataset Summary

PSVBench is an evaluation benchmark for question answering over academic and educational videos.
It is designed for benchmark evaluation only, not for training.

Each example in `qa/eval.json` contains:

- a question ID
- a task label
- a sub-task label
- a video path
- a transcript path
- a gold temporal window
- a 4-way multiple-choice question
- the correct answer

The benchmark covers content from multiple domains and sources, including:
- biology
- math
- chemistry
- physics
- medicine
- NeurIPS
- ICML
- ICLR
- CVPR
- ECCV
- ACL

Some samples also contain slide images in their media directories.

## Repository Structure
```text
psvbench/
├── README.md
├── qa/
│ └── eval.json
└── data/
├── biology/
├── math/
├── chemistry/
├── physics/
├── medicine/
├── neurips/
├── icml/
├── iclr/
├── cvpr/
├── eccv/
└── acl/
```
All paths in qa/eval.json are relative to the repository root.

## Data Format
qa/eval.json is a JSON list. Each record has the following fields:
- id
- task
- sub_task
- video_path
- transcript_path
- gold_window
- question
- options
- answer
Example:
```text
{
"id": "PLE_QA_244",
"task": "Understanding",
"sub_task": "Procedural Logic Evolution",
"video_path": "data/math/Metric Spaces - Lectures 19 & 20_ Oxford Mathematics 2nd Year Student Lecture/Metric Spaces -
Lectures 19 & 20_ Oxford Mathematics 2nd Year Student Lecture.mp4",
"transcript_path": "data/math/Metric Spaces - Lectures 19 & 20_ Oxford Mathematics 2nd Year Student Lecture/transcript/Metric
Spaces - Lectures 19 & 20_ Oxford Mathematics 2nd Year Student Lecture.vtt",
"gold_window": [697.26, 725.1],
"question": "What is the next logical step after proving that the projection maps preserve limits?",
"options": {
"A": "Conclude that x_n converges to a and y_n converges to b.",
"B": "Introduce a new metric for the product space.",
"C": "Prove that the projection maps are Lipschitz continuous.",
"D": "Compute the distance between (x_n, y_n) and (a, b)."
},
"answer": "D"
}
```

## Limitations

- This dataset is intended for evaluation only.
- Slide materials are available only for a subset of samples.
- Users are responsible for checking upstream source terms and usage restrictions.

## License

Please check the original source licenses and usage restrictions.

Where upstream content has additional restrictions, those restrictions take precedence.