s1-vis-mid-resize

# s1-vis-mid-resize

Original dataset structure preserved, filtered by token length and image quality

## Dataset Description

This dataset was processed using the [data-preproc](https://github.com/oumi-ai/ml-preproc) package for vision-language model training.

### Processing Configuration

- **Base Model**: Qwen/Qwen2.5-7B-Instruct
- **Tokenizer**: Qwen/Qwen2.5-7B-Instruct
- **Sequence Length**: 16384
- **Processing Type**: Vision Language (VL)

### Dataset Features

- **input_ids**: Tokenized input sequences
- **attention_mask**: Attention masks for the sequences
- **labels**: Labels for language modeling
- **images**: PIL Image objects
- **messages**: Original conversation messages
- **metadata**: Processing metadata

### Processing Statistics

- **Original Samples**: 812
- **Processed Samples**: 812
- **Success Rate**: 100.0%
- **Average Token Length**: N/A
- **Max Token Length**: N/A
- **Truncation Rate**: N/A

### Usage

```python
from datasets import load_dataset

# Load the dataset
dataset = load_dataset("your-org/your-dataset-name")

# Access samples
sample = dataset["train"][0]
print(f"Input tokens: {len(sample['input_ids'])}")
print(f"Images: {len(sample['images'])}")
print(f"Messages: {sample['messages']}")
```

## License

This dataset is released under the specified license. Please check the license field for details.

# s1-vis-mid-resize
保留原始数据集结构，基于词元(Token)长度与图像质量完成筛选

## 数据集描述
本数据集通过[data-preproc](https://github.com/oumi-ai/ml-preproc)工具包处理，用于视觉语言模型训练。

### 处理配置
- **基础模型**：Qwen/Qwen2.5-7B-Instruct
- **分词器**：Qwen/Qwen2.5-7B-Instruct
- **序列长度**：16384
- **处理类型**：视觉语言(Vision Language, VL)

### 数据集特征
- **input_ids**：已分词的输入词元序列
- **attention_mask**：序列注意力掩码
- **labels**：语言建模任务标签
- **images**：PIL图像对象
- **messages**：原始对话消息
- **metadata**：处理元数据

### 处理统计数据
- **原始样本量**：812
- **处理后样本量**：812
- **处理成功率**：100.0%
- **平均词元长度**：无可用数据
- **最大词元长度**：无可用数据
- **截断率**：无可用数据

### 使用方法
python
from datasets import load_dataset

# 加载数据集
dataset = load_dataset("your-org/your-dataset-name")

# 访问样本
sample = dataset["train"][0]
print(f"输入词元数：{len(sample['input_ids'])}")
print(f"图像数量：{len(sample['images'])}")
print(f"对话消息：{sample['messages']}")

## 许可证
本数据集已按指定许可证发布，详情请查看许可证字段。