pixmo-points

# PixMo-Points
PixMo-Points is a dataset of images paired with referring expressions and points marking the locations the
referring expression refers to in the image. It was collected using human annotators and contains a diverse
range of points and expressions, with many high-frequency (10+) expressions.

PixMo-Points is a part of the [PixMo dataset collection](https://huggingface.co/collections/allenai/pixmo-674746ea613028006285687b) and was used to
provide the pointing capabilities of the [Molmo family of models](https://huggingface.co/collections/allenai/molmo-66f379e6fe3b8ef090a8ca19)

Quick links:
- 📃 [Paper](https://molmo.allenai.org/paper.pdf)
- 🎥 [Blog with Videos](https://molmo.allenai.org/blog)

## Loading
```python
data = datasets.load_dataset("allenai/pixmo-points", split="train")
```

## Data Format
Images are stored as URLs that will need to be downloaded separately. Note URLs can be repeated in the data.

The `points` field contains the x, y coordinates specified in pixels.

The `label` field contains the string name of what is being pointed at, this can be a simple object name or a more complex referring expression.

The `collection_method` field specifies whether the image was chosen to target high-frequency counting ("counting") or general pointing ("pointing").

## Image Checking
Image hashes are included to support double-checking that the downloaded image matches the annotated image.
It can be checked like this:

```python
from hashlib import sha256
import requests

example = data[0]
image_bytes = requests.get(example["image_url"]).content
byte_hash = sha256(image_bytes).hexdigest()
assert byte_hash == example["image_sha256"]
```

## License
This dataset is licensed under ODC-BY-1.0. It is intended for research and educational use in accordance with Ai2's [Responsible Use Guidelines](https://allenai.org/responsible-use).

# PixMo-Points 数据集
PixMo-Points是一类将图像与指代表达式（referring expression）以及标注该表达式在图像中指向位置的点进行配对的数据集。本数据集通过人工标注人员采集，涵盖了丰富多样的点标注与指代表达式，其中包含大量出现频次不低于10次的高频表达式。

PixMo-Points隶属于[PixMo数据集合集](https://huggingface.co/collections/allenai/pixmo-674746ea613028006285687b)，曾用于为[Molmo系列模型](https://huggingface.co/collections/allenai/molmo-66f379e6fe3b8ef090a8ca19)提供点指向交互能力。

## 快速链接
- 📃 [论文](https://molmo.allenai.org/paper.pdf)
- 🎥 [含演示视频的博客](https://molmo.allenai.org/blog)

## 数据集加载
python
data = datasets.load_dataset("allenai/pixmo-points", split="train")

## 数据格式
图像以URL形式存储，需单独下载。需注意，数据中可能存在重复的URL。

`points`字段存储以像素为单位的x、y坐标。

`label`字段存储被指向目标的字符串名称，既可以是简单的物体名称，也可以是更为复杂的指代表达式。

`collection_method`字段用于指定图像的采集目标：是针对高频计数任务（"counting"）还是通用点指向任务（"pointing"）。

## 图像校验
数据中包含图像哈希值，用于校验下载得到的图像与标注图像是否一致。校验代码示例如下：
python
from hashlib import sha256
import requests

example = data[0]
image_bytes = requests.get(example["image_url"]).content
byte_hash = sha256(image_bytes).hexdigest()
assert byte_hash == example["image_sha256"]

## 许可证
本数据集采用ODC-BY-1.0许可证进行授权，仅可用于科研与教育用途，并需遵循AI2的[负责任使用指南](https://allenai.org/responsible-use).

搜集汇总

背景与挑战

背景概述

PixMo-Points是一个图像与指代表达式及对应标记点位置配对的数据集，用于支持Molmo系列模型的指向能力。数据集包含多样化的点和表达，提供图像URL、坐标点、标签和收集方法等信息，并包含图像哈希值用于验证。