coco-multilabel

## 数据集

数据集地址：https://modelscope.cn/datasets/swift/coco-multilabel
数据集下载：
```python
from modelscope import MsDataset
dataset = MsDataset.load('swift/coco-multilabel', split='train')
test_dataset = MsDataset.load('swift/coco-multilabel', split='test')
print(dataset)
print(test_dataset)
"""
Dataset({
features: ['images', 'labels'],
num_rows: 117218
})
Dataset({
features: ['images', 'labels'],
num_rows: 5000
})
"""
print(dataset[0].keys(), dataset[0]['labels'])
print(test_dataset[0].keys(), test_dataset[0]['labels'])
"""
dict_keys(['images', 'labels']) [49, 76]
dict_keys(['images', 'labels']) None
"""
# 查看图片
from PIL import Image
import io
image_bytes = dataset[0]['images'][0]['bytes']
image = Image.open(io.BytesIO(image_bytes))
image.save('image.png')
```

该数据集包含 117,218 条训练样本 和 5,000 条测试样本。在训练集中，每条样本均包含真实标签 labels，代表图像中包含哪些东西（例如：person, cat, train等）。测试集中则不包含 labels 标签，模型需要根据测试集中提供 "images" 字段，推断出labels是什么，本质是多标签分类问题。你可以使用生成式任务/多标签任务来解决这个问题。

标签的对应关系请查看`labels.txt`文件。

## Baseline

以下介绍使用ms-swift大模型训练框架，对Qwen3.5-4B使用该数据集进行LoRA微调的baseline
- ms-swift github：https://github.com/modelscope/ms-swift
- Qwen3.5: https://modelscope.cn/models/Qwen/Qwen3.5-4B
- 以下提供的baseline所需显存资源12GiB，可在魔搭免费算力A10上运行
- 该baseline的F1为: 0.8925

环境准备：
```shell
pip install "ms-swift==4.0.*" "transformers==5.2.*" "peft==0.18.*" -U
```

单卡训练：
```python
# GPU Memory: 12GB
import os
from typing import Any, Dict

from swift import SftArguments, sft_main
from swift.dataset import DatasetMeta, ResponsePreprocessor, SubsetDataset, register_dataset

os.environ['CUDA_VISIBLE_DEVICES'] = '0'
os.environ['MAX_PIXELS'] = str(1024 * 32 * 32)

labels = [
'airplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle',
'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair',
'clock', 'couch', 'cow', 'cup', 'dining table', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee',
'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave',
'motorcycle', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'potted plant', 'refrigerator',
'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'spoon', 'sports ball',
'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush',
'traffic light', 'train', 'truck', 'tv', 'umbrella', 'vase', 'wine glass', 'zebra'
]
labels_str = ', '.join(labels)

class CustomPreprocessor(ResponsePreprocessor):

def preprocess(self, row: Dict[str, Any]) -> Dict[str, Any]:
row['query'] = f'Multi-label classification with the following categories: {labels_str}'
return super().preprocess(row)

register_dataset(
DatasetMeta(
ms_dataset_id='swift/coco-multilabel',
preprocess_func=CustomPreprocessor(columns={'labels': 'label'}),
subsets=[SubsetDataset('train', split=['train']),
SubsetDataset('test', split=['test'])]))

if __name__ == '__main__':
sft_main(
SftArguments(
model='Qwen/Qwen3.5-4B',
tuner_type='lora',
# 节约时间，只选择20000条数据集
dataset=['swift/coco-multilabel:train#20000'],
load_from_cache_file=True,
add_non_thinking_prefix=True,
loss_scale='ignore_empty_think',
split_dataset_ratio=0.01,
torch_dtype='bfloat16',
num_train_epochs=1,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
learning_rate=1e-4,
lora_rank=8,
lora_alpha=32,
target_modules=['all-linear'],
freeze_vit=True,
gradient_accumulation_steps=4,
eval_steps=100,
save_steps=100,
save_total_limit=2,
logging_steps=5,
max_length=2048,
output_dir='output',
warmup_ratio=0.05,
dataset_num_proc=4,
dataloader_num_workers=4,
num_labels=80,
task_type='seq_cls',
problem_type='multi_label_classification',
))
```

## 提交结果

我们提供了推理脚本, 最终需要将以下推理脚本产生的`infer_result`目录中的jsonl文件进行提交 (由于比赛界面只允许传递json后缀的文件, 请重命名为`result.json`, 不需要改内容).

```python
# GPU Memory: 12GB
import os
from typing import Any, Dict

from swift import InferArguments, infer_main
from swift.dataset import DatasetMeta, ResponsePreprocessor, SubsetDataset, register_dataset

os.environ['CUDA_VISIBLE_DEVICES'] = '0'
os.environ['MAX_PIXELS'] = str(1024 * 32 * 32)

labels = [
'airplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle',
'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair',
'clock', 'couch', 'cow', 'cup', 'dining table', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee',
'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave',
'motorcycle', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'potted plant', 'refrigerator',
'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'spoon', 'sports ball',
'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush',
'traffic light', 'train', 'truck', 'tv', 'umbrella', 'vase', 'wine glass', 'zebra'
]
labels_str = ', '.join(labels)

class CustomPreprocessor(ResponsePreprocessor):

def preprocess(self, row: Dict[str, Any]) -> Dict[str, Any]:
row['query'] = f'Multi-label classification with the following categories: {labels_str}'
return super().preprocess(row)

register_dataset(
DatasetMeta(
ms_dataset_id='swift/coco-multilabel',
preprocess_func=CustomPreprocessor(columns={'labels': 'label'}),
subsets=[SubsetDataset('train', split=['train']),
SubsetDataset('test', split=['test'])]))

if __name__ == '__main__':
ckpt_dir = 'output/vx-xxx/checkpoint-xxx' # last_checkpoint
result = infer_main(InferArguments(
adapters=[ckpt_dir],
temperature=0,
val_dataset=["swift/coco-multilabel:test"],
max_batch_size=16,
infer_backend='transformers'))
# 结果会保存在`{ckpt_dir}/infer_result/xxx-xxx.jsonl`中, 提交该文件即可.
# (由于比赛界面只能传递json后缀的文件, 请重命名为`result.json`, 不需要改内容).
```

提交的jsonl文件格式如下，顺序与`test_dataset`顺序一致，共5000条，
```
{"response": [7, 22, 24, 64]}
{"response": [...]}
{"response": [...]}
```

打分脚本：

最终结果以F1为评测标准进行排序。

```python
from swift.utils import read_from_jsonl
from datasets import load_dataset

labels = load_dataset('parquet', data_files='test_labels.parquet', split='train')['labels']

results = read_from_jsonl('result.jsonl')

tp, fp, fn = 0, 0, 0

for i, (res, label) in enumerate(zip(results, labels)):
y_pred = set(res['response'])
y_true = set(label)
intersection = y_pred & y_true

tp += len(intersection)
fp += len(y_pred - y_true)
fn += len(y_true - y_pred)

precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)

print(f'Precision: {precision:.4f}')
print(f'Recall: {recall:.4f}')
print(f'F1 Score: {f1:.4f}')
```

## 数据集

数据集地址：https://modelscope.cn/datasets/swift/coco-multilabel
数据集下载：
python
from modelscope import MsDataset
dataset = MsDataset.load('swift/coco-multilabel', split='train')
test_dataset = MsDataset.load('swift/coco-multilabel', split='test')
print(dataset)
print(test_dataset)
"""
Dataset({
features: ['images', 'labels'],
num_rows: 117218
})
Dataset({
features: ['images', 'labels'],
num_rows: 5000
})
"""
print(dataset[0].keys(), dataset[0]['labels'])
print(test_dataset[0].keys(), test_dataset[0]['labels'])
"""
dict_keys(['images', 'labels']) [49, 76]
dict_keys(['images', 'labels']) None
"""
# 查看图片
from PIL import Image
import io
image_bytes = dataset[0]['images'][0]['bytes']
image = Image.open(io.BytesIO(image_bytes))
image.save('image.png')

该数据集共包含117218条训练样本与5000条测试样本。训练集中每条样本均带有真实标签`labels`，用于表征图像中包含的物体类别（例如person、cat、train等）；测试集则未提供`labels`标签，模型需根据测试集的`images`字段推断对应的标签，本质为多标签分类任务，可通过生成式任务或多标签任务完成求解。标签与编号的对应关系请查看`labels.txt`文件。

## 基准方案

以下介绍使用ms-swift大模型训练框架，对通义千问3.5-4B（Qwen3.5-4B）基于该数据集进行LoRA微调的基准方案：
- ms-swift开源仓库：https://github.com/modelscope/ms-swift
- 通义千问3.5-4B（Qwen3.5-4B）模型地址：https://modelscope.cn/models/Qwen/Qwen3.5-4B
- 本基准方案所需显存为12GiB，可在魔搭社区（ModelScope）免费算力A10上运行
- 该基准方案的F1值为0.8925

环境准备：
shell
pip install "ms-swift==4.0.*" "transformers==5.2.*" "peft==0.18.*" -U

单卡训练：
python
# 显存占用：12GB
import os
from typing import Any, Dict

from swift import SftArguments, sft_main
from swift.dataset import DatasetMeta, ResponsePreprocessor, SubsetDataset, register_dataset

os.environ['CUDA_VISIBLE_DEVICES'] = '0'
os.environ['MAX_PIXELS'] = str(1024 * 32 * 32)

labels = [
'airplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle',
'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair',
'clock', 'couch', 'cow', 'cup', 'dining table', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee',
'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave',
'motorcycle', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'potted plant', 'refrigerator',
'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'spoon', 'sports ball',
'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush',
'traffic light', 'train', 'truck', 'tv', 'umbrella', 'vase', 'wine glass', 'zebra'
]
labels_str = ', '.join(labels)

class CustomPreprocessor(ResponsePreprocessor):

def preprocess(self, row: Dict[str, Any]) -> Dict[str, Any]:
row['query'] = f'Multi-label classification with the following categories: {labels_str}'
return super().preprocess(row)

register_dataset(
DatasetMeta(
ms_dataset_id='swift/coco-multilabel',
preprocess_func=CustomPreprocessor(columns={'labels': 'label'}),
subsets=[SubsetDataset('train', split=['train']),
SubsetDataset('test', split=['test'])]))

if __name__ == '__main__':
sft_main(
SftArguments(
model='Qwen/Qwen3.5-4B',
tuner_type='lora',
# 节约时间，只选择20000条数据集
dataset=['swift/coco-multilabel:train#20000'],
load_from_cache_file=True,
add_non_thinking_prefix=True,
loss_scale='ignore_empty_think',
split_dataset_ratio=0.01,
torch_dtype='bfloat16',
num_train_epochs=1,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
learning_rate=1e-4,
lora_rank=8,
lora_alpha=32,
target_modules=['all-linear'],
freeze_vit=True,
gradient_accumulation_steps=4,
eval_steps=100,
save_steps=100,
save_total_limit=2,
logging_steps=5,
max_length=2048,
output_dir='output',
warmup_ratio=0.05,
dataset_num_proc=4,
dataloader_num_workers=4,
num_labels=80,
task_type='seq_cls',
problem_type='multi_label_classification',
))

## 提交方式

我们提供了推理脚本，最终需将以下推理脚本生成的`infer_result`目录内的JSON行文件（jsonl）提交。由于比赛界面仅支持上传JSON后缀文件，请将其重命名为`result.json`，无需修改内容。

python
# 显存占用：12GB
import os
from typing import Any, Dict

from swift import InferArguments, infer_main
from swift.dataset import DatasetMeta, ResponsePreprocessor, SubsetDataset, register_dataset

os.environ['CUDA_VISIBLE_DEVICES'] = '0'
os.environ['MAX_PIXELS'] = str(1024 * 32 * 32)

labels = [
'airplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle',
'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair',
'clock', 'couch', 'cow', 'cup', 'dining table', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee',
'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave',
'motorcycle', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'potted plant', 'refrigerator',
'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'spoon', 'sports ball',
'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush',
'traffic light', 'train', 'truck', 'tv', 'umbrella', 'vase', 'wine glass', 'zebra'
]
labels_str = ', '.join(labels)

class CustomPreprocessor(ResponsePreprocessor):

def preprocess(self, row: Dict[str, Any]) -> Dict[str, Any]:
row['query'] = f'Multi-label classification with the following categories: {labels_str}'
return super().preprocess(row)

register_dataset(
DatasetMeta(
ms_dataset_id='swift/coco-multilabel',
preprocess_func=CustomPreprocessor(columns={'labels': 'label'}),
subsets=[SubsetDataset('train', split=['train']),
SubsetDataset('test', split=['test'])]))

if __name__ == '__main__':
ckpt_dir = 'output/vx-xxx/checkpoint-xxx' # last_checkpoint
result = infer_main(InferArguments(
adapters=[ckpt_dir],
temperature=0,
val_dataset=["swift/coco-multilabel:test"],
max_batch_size=16,
infer_backend='transformers'))
# 推理结果将保存至`{ckpt_dir}/infer_result/xxx-xxx.jsonl`，提交该文件即可。由于比赛界面仅支持上传JSON后缀文件，请将其重命名为`result.json`，无需修改内容。

提交的JSON行文件格式如下，顺序与`test_dataset`一致，共5000条：
jsonl
{"response": [7, 22, 24, 64]}
{"response": [...]}
{"response": [...]}

## 评分脚本

最终结果以F1值作为评测标准进行排序。

python
from swift.utils import read_from_jsonl
from datasets import load_dataset

labels = load_dataset('parquet', data_files='test_labels.parquet', split='train')['labels']

results = read_from_jsonl('result.jsonl')

tp, fp, fn = 0, 0, 0

for i, (res, label) in enumerate(zip(results, labels)):
y_pred = set(res['response'])
y_true = set(label)
intersection = y_pred & y_true

tp += len(intersection)
fp += len(y_pred - y_true)
fn += len(y_true - y_pred)

precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)

print(f'Precision: {precision:.4f}')
print(f'Recall: {recall:.4f}')
print(f'F1 Score: {f1:.4f}')