konpat/visual-jenga-datasets

---
language:
- en
pretty_name: Visual Jenga Datasets
tags:
- scene-understanding
- object-removal
- computer-vision
- scene-graph
- coco
- nyu-depth
task_categories:
- image-segmentation
- object-detection
- mask-generation
size_categories:
- 1K<n<10K
license: other
---

# Visual Jenga Datasets

This directory contains the original datasets for [Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting](https://visualjenga.github.io/). Visual Jenga is a novel scene understanding task that involves progressively removing objects from a single image one at a time while keeping the rest of the scene stable. This process reveals object dependencies and provides a new way to evaluate grounded scene understanding by systematically exploring which objects can be removed while preserving scene coherence in both physical and geometric sense.

## Datasets Overview

### 1. ClutteredParse
- **Directory**: `clutteredparse/`
- **Samples**: 40 scenes
- **Structure**: Each subdirectory contains:
- `img.*` - Original scene image (various formats: jpg, jpeg, webp)
- `A.png` - Mask showing objects to remove (before)
- `B.png` - Mask showing objects after removal (after)
- **Visualization**: [dataset_clutteredparse.html](dataset_clutteredparse.html)

### 2. COCO
- **Directory**: `coco/`
- **Samples**: 200 scenes
- **Structure**: Each subdirectory (numbered 000-199) contains:
- `img.*` - Original scene image (jpeg format)
- `A.png` - Mask showing objects to remove (before)
- `B.png` - Mask showing objects after removal (after)
- **Visualization**: [dataset_coco.html](dataset_coco.html)

### 3. Full Scene Decom
- **Directory**: `full_scene_decom/`
- **Samples**: 56 images
- **Structure**: Flat directory with sequentially numbered images (00-55) in various formats (jpg, png, jpeg, webp, avif)
- **Purpose**: Scene decomposition visualization showing progressive scene breakdown
- **Visualization**: [dataset_full_scene_decom.html](dataset_full_scene_decom.html)

### 4. NYU
- **Directory**: `nyu/`
- **Samples**: 668 scenes
- **Structure**: Each subdirectory contains:
- `img.*` - Original scene image (jpg format)
- `A.png` - Mask showing objects to remove (before)
- `B.png` - Mask showing objects after removal (after)
- `scene_graph.jpg` or `scene_graph.png` - Scene graph visualization
- **Visualization**: [dataset_nyu.html](dataset_nyu.html)

## File Structure

```
originals/
├── clutteredparse/ # ClutteredParse dataset
│ ├── 01/
│ │ ├── img.jpg
│ │ ├── A.png
│ │ └── B.png
│ ├── 02/
│ └── ...
├── coco/ # COCO dataset
│ ├── 000/
│ │ ├── img.jpeg
│ │ ├── A.png
│ │ └── B.png
│ ├── 001/
│ └── ...
├── full_scene_decom/ # Full Scene Decom dataset
│ ├── 00.jpg
│ ├── 01.png
│ └── ...
├── nyu/ # NYU dataset
│ ├── 10_0/
│ │ ├── img.jpg
│ │ ├── A.png
│ │ ├── B.png
│ │ └── scene_graph.jpg
│ └── ...
├── dataset_clutteredparse.html
├── dataset_coco.html
├── dataset_full_scene_decom.html
├── dataset_nyu.html
└── README.md
```

## Visualizations

Interactive HTML visualizations are available for each dataset:

- **[ClutteredParse Visualization](dataset_clutteredparse.html)** - Browse all 40 scenes with before/after masks
- **[COCO Visualization](dataset_coco.html)** - Browse all 200 scenes with before/after masks
- **[Full Scene Decom Visualization](dataset_full_scene_decom.html)** - Browse all 56 scene decomposition images
- **[NYU Visualization](dataset_nyu.html)** - Browse all 668 scenes with before/after masks and scene graphs

## Dataset Statistics

| Dataset | Scenes | Total Files | Image Formats |
|---------|--------|-------------|---------------|
| ClutteredParse | 40 | 120 | jpg, jpeg, webp, png |
| COCO | 200 | 600 | jpeg, png |
| Full Scene Decom | 56 | 56 | jpg, png, jpeg, webp, avif |
| NYU | 668 | 2,672 | jpg, png |

## License

This dataset combines images and annotations from multiple sources with different licensing terms. Users must comply with the original licenses of each source:

### Source Licenses

1. **COCO Dataset**
- Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/)
- **Reference**: [COCO Dataset](https://cocodataset.org/)

2. **NYU Depth V2 Dataset**
- **Reference**: [NYU Depth Dataset V2](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html)
- **Citation**: Please cite: "Indoor Segmentation and Support Inference from RGBD Images" (ECCV 2012) by Silberman et al.

3. **ClutteredParse & Full Scene Decom**
- Images sourced from the internet with no guaranteed license
- These images may be subject to copyright protection

**Disclaimer**: This dataset is provided "as-is" without warranty. Users are solely responsible for ensuring their use complies with all applicable licenses and copyright laws.