mrdbourke/food-drink-items-1B

--- dataset_info: features: - name: item dtype: string - name: item_type dtype: string - name: count_re dtype: int64 - name: count_org dtype: int64 - name: count_total dtype: int64 - name: selection dtype: string - name: is_human_edible dtype: bool - name: is_generic_label dtype: bool - name: is_branded_item dtype: bool - name: is_raw_ingredient dtype: bool - name: is_dish dtype: bool - name: is_container_or_utensil dtype: bool - name: food_categories dtype: string - name: canonical_form dtype: string splits: - name: train num_examples: 292783 configs: - config_name: default data_files: - split: train path: data/train/*.parquet license: apache-2.0 task_categories: - text-classification - zero-shot-classification language: - en tags: - food - drink - food-classification - food-extraction - food-vocabulary - datacomp - billion-scale size_categories: - 100K<n<1M source_datasets: - UCSC-VLAA/Recap-DataComp-1B --- # Food & Drink Items from 1 Billion Image Captions A structured vocabulary of **292,783 unique food and drink items** extracted from [UCSC-VLAA/Recap-DataComp-1B](https://huggingface.co/datasets/UCSC-VLAA/Recap-DataComp-1B) (1 billion image captions), enriched with multi-label category tags and edibility classification. ## Overview | | Count | Percentage | |---|---|---| | **Total items** | 292,783 | 100% | | **Human edible** | 218,293 | 74.6% | | **Non-edible** | 74,490 | 25.4% | | **Food items** | 228,504 | | | **Drink items** | 64,279 | | | **Branded items** | 80,143 | | | **Dishes** | 73,950 | | | **Raw ingredients** | 38,195 | | | **Containers/utensils** | 12,091 | | | **Unique canonical forms** | 277,249 | | | **Total mentions across 1B captions** | 253,522,547 | | ## How it was made 1. **Source:** [UCSC-VLAA/Recap-DataComp-1B](https://huggingface.co/datasets/UCSC-VLAA/Recap-DataComp-1B) — 1 billion image-caption pairs 2. **Text classification:** [mrdbourke/Recap-DataComp-1B-FoodOrDrink](https://huggingface.co/datasets/mrdbourke/Recap-DataComp-1B-FoodOrDrink) — ~106M rows classified as food/drink 3. **Food extraction:** [mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2](https://huggingface.co/mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2) — extracted specific food/drink item names from both `re_caption` (AI-generated) and `org_caption` (web alt-text) 4. **Counting:** Item frequencies computed across all 106M rows from both caption types 5. **Enrichment:** Each item classified by [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) with structured metadata (edibility, category tags, canonical form) ### Item selection | Selection | Count | Description | |---|---|---| | `threshold` | 95,174 | All items appearing >= 100 times (covers ~84% of all mentions) | | `random_sample` | 197,609 | 1% random sample of items below threshold (long-tail coverage) | ## Usage ```python from datasets import load_dataset import json ds = load_dataset("mrdbourke/food-drink-items-1B", split="train") print(f"Total items: {len(ds):,}") # All human-edible items sorted by frequency edible = ds.filter(lambda x: x["is_human_edible"]).sort("count_total", reverse=True) print(f"Edible items: {len(edible):,}") # Parse multi-label categories row = ds[0] categories = json.loads(row["food_categories"]) print(f"{row['item']}: {categories}") # Find all dishes dishes = ds.filter(lambda x: '"dish"' in x["food_categories"]) # Find all seafood items seafood = ds.filter(lambda x: '"seafood"' in x["food_categories"]) # Get branded items brands = ds.filter(lambda x: x["is_branded_item"]) # Non-food items (useful as training negatives) non_food = ds.filter(lambda x: not x["is_human_edible"]) # Deduplicate by canonical form (merges "tomatoes" + "Tomatoes" + "tomato") from collections import defaultdict canonical = defaultdict(int) for row in ds: canonical[row["canonical_form"]] += row["count_total"] top_100 = sorted(canonical.items(), key=lambda x: -x[1])[:100] ``` ## Fields | Field | Type | Description | |---|---|---| | `item` | string | Original extracted item name (normalized to lowercase) | | `item_type` | string | `"food"` or `"drink"` — which extraction list it came from | | `count_re` | int | Number of times this item was extracted from `re_caption` (AI-generated captions) | | `count_org` | int | Number of times this item was extracted from `org_caption` (web alt-text) | | `count_total` | int | `count_re + count_org` | | `selection` | string | `"threshold"` (count >= 100) or `"random_sample"` (1% of long tail) | | `is_human_edible` | bool | Would a person eat or drink this? | | `is_generic_label` | bool | Is this a vague descriptor? ("red liquid", "food", "beverage") | | `is_branded_item` | bool | Is this a brand name? ("Coca-Cola", "Jack Daniel's") | | `is_raw_ingredient` | bool | Single whole/unprocessed ingredient? ("apple" = true, "apple pie" = false) | | `is_dish` | bool | Prepared/composed dish or recipe? ("pad thai" = true, "rice" = false) | | `is_container_or_utensil` | bool | Container, vessel, or utensil? ("wine glass", "mug") | | `food_categories` | string (JSON list) | Multi-label category tags (see below) | | `canonical_form` | string | Normalized: lowercase, singular, stripped | ## Category tags The `food_categories` field contains a JSON-encoded list of one or more tags. An item like "shrimp pad thai" would have `["dish", "seafood", "grain"]`. | Category | Count | % of items | |---|---|---| | `non_food` | 68,955 | 23.6% | | `dish` | 56,216 | 19.2% | | `drink` | 45,314 | 15.5% | | `confectionary` | 32,440 | 11.1% | | `baked_goods` | 30,878 | 10.5% | | `meat` | 23,899 | 8.2% | | `vegetable` | 23,113 | 7.9% | | `fruit` | 21,802 | 7.4% | | `grain` | 21,735 | 7.4% | | `liquor` | 21,495 | 7.3% | | `dairy` | 20,268 | 6.9% | | `snack` | 16,267 | 5.6% | | `other` | 15,718 | 5.4% | | `condiments` | 15,471 | 5.3% | | `supplement` | 9,374 | 3.2% | | `seafood` | 9,023 | 3.1% | | `additive` | 8,731 | 3.0% | | `herbs_and_spices` | 7,106 | 2.4% | | `nuts_and_seeds` | 6,097 | 2.1% | | `sweetener` | 5,020 | 1.7% | | `frozen_dessert` | 4,868 | 1.7% | | `legume` | 4,377 | 1.5% | | `spread` | 3,522 | 1.2% | | `side_dish` | 2,297 | 0.8% | | `eggs` | 2,291 | 0.8% | | `oil` | 2,026 | 0.7% | | `pet_food` | 1,678 | 0.6% | | `cereals` | 1,466 | 0.5% | | `fungi` | 1,368 | 0.5% | ## Caption types Items were extracted from two different caption types, reflected in the count columns: - **`count_re`** — from `re_caption`: AI-generated detailed captions (LLaVA-1.5-LLaMA3-8B). Produces generic visual descriptions ("meat", "vegetables", "sauce") - **`count_org`** — from `org_caption`: Original web-crawled alt-text. Produces specific named items ("wagyu ribeye", "pad thai", "marinara sauce") Using both gives complementary coverage. High `count_re` items are visually common; high `count_org` items are frequently named on the web. ## Models used | Stage | Model | Purpose | |---|---|---| | Text classification | [mrdbourke/ettin-150m-food-or-drink-classifier](https://huggingface.co/mrdbourke/ettin-150m-food-or-drink-classifier) | Binary food/not-food on 1B captions | | Food extraction | [mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2](https://huggingface.co/mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2) | Extract item names + tags from captions | | Item enrichment | [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | Classify items with structured metadata | ## Limitations - Item names are extracted by a 270M parameter model and may include errors - Enrichment labels are from a 9B parameter model, not human-annotated - Long-tail items (count < 100) are sampled at 1%, not exhaustive - `food_categories` is stored as a JSON string, not a native list type - Some canonical forms may not perfectly deduplicate (e.g. regional spellings) - Counts reflect caption frequency, not real-world food popularity ## License Apache 2.0 — consistent with source dataset and all models used. ## Source - **Source dataset**: [UCSC-VLAA/Recap-DataComp-1B](https://huggingface.co/datasets/UCSC-VLAA/Recap-DataComp-1B) - **Filtered dataset**: [mrdbourke/Recap-DataComp-1B-FoodOrDrink](https://huggingface.co/datasets/mrdbourke/Recap-DataComp-1B-FoodOrDrink) - **FoodExtract model**: [mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2](https://huggingface.co/mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2) - **Enrichment model**: [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)数据集信息： 特征： - 名称：item 数据类型：字符串（string） - 名称：item_type 数据类型：字符串（string） - 名称：count_re 数据类型：64位整数（int64） - 名称：count_org 数据类型：64位整数（int64） - 名称：count_total 数据类型：64位整数（int64） - 名称：selection 数据类型：字符串（string） - 名称：is_human_edible 数据类型：布尔型（bool） - 名称：is_generic_label 数据类型：布尔型（bool） - 名称：is_branded_item 数据类型：布尔型（bool） - 名称：is_raw_ingredient 数据类型：布尔型（bool） - 名称：is_dish 数据类型：布尔型（bool） - 名称：is_container_or_utensil 数据类型：布尔型（bool） - 名称：food_categories 数据类型：字符串（string） - 名称：canonical_form 数据类型：字符串（string） 数据集划分： - 名称：train 样本数：292783 配置项： - 配置名称：default 数据文件： - 划分：train 路径：data/train/*.parquet 许可证：Apache-2.0 任务类别： - 文本分类 - 零样本（Zero-shot）分类 语言： - 英语 标签： - 食品 - 饮品 - 食品分类 - 食品提取 - 食品词汇表 - DataComp - 十亿级规模 规模类别： - 100K<n<1M 源数据集： - UCSC-VLAA/Recap-DataComp-1B # 来自10亿图像字幕的食品与饮品条目 本数据集包含从[UCSC-VLAA/Recap-DataComp-1B](https://huggingface.co/datasets/UCSC-VLAA/Recap-DataComp-1B)（含10亿条图像字幕）中提取的**292783个唯一食品与饮品条目**的结构化词汇集，辅以多标签分类标签与可食用性分类信息。 ## 数据集概览 | | 数量 | 占比 | |---|---|---| | **总条目数** | 292783 | 100% | | **人类可食用** | 218293 | 74.6% | | **非食用类** | 74490 | 25.4% | | **食品条目** | 228504 | | | **饮品条目** | 64279 | | | **品牌条目** | 80143 | | | **菜肴** | 73950 | | | **原料食材** | 38195 | | | **容器/器具** | 12091 | | | **唯一规范形式** | 277249 | | | **10亿字幕总提及量** | 253522547 | | ## 数据集构建流程 1. **数据源**：[UCSC-VLAA/Recap-DataComp-1B](https://huggingface.co/datasets/UCSC-VLAA/Recap-DataComp-1B) — 10亿条图像-字幕配对数据 2. **文本分类**：[mrdbourke/Recap-DataComp-1B-FoodOrDrink](https://huggingface.co/datasets/mrdbourke/Recap-DataComp-1B-FoodOrDrink) — 对约1.06亿条数据进行食品/饮品分类 3. **食品提取**：使用[mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2](https://huggingface.co/mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2)模型，从`re_caption`（AI生成字幕）与`org_caption`（网页替代文本）中提取具体的食品/饮品条目名称 4. **计数统计**：计算两种字幕来源中所有1.06亿条数据内的条目出现频率 5. **元数据富集**：使用[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)模型为每个条目添加结构化元数据，包括可食用性、分类标签与规范形式 ### 条目筛选规则 | 筛选方式 | 数量 | 说明 | |---|---|---| | `threshold`（阈值筛选） | 95174 | 所有出现次数≥100的条目（覆盖约84%的总提及量） | | `random_sample`（随机采样） | 197609 | 对低于阈值的条目进行1%随机采样（覆盖长尾数据） | ## 使用示例 python from datasets import load_dataset import json ds = load_dataset("mrdbourke/food-drink-items-1B", split="train") print(f"总条目数: {len(ds):,}") # 按出现频率降序排列的所有人类可食用条目 edible = ds.filter(lambda x: x["is_human_edible"]).sort("count_total", reverse=True) print(f"可食用条目数: {len(edible):,}") # 解析多标签分类 row = ds[0] categories = json.loads(row["food_categories"]) print(f"{row['item']}: {categories}") # 筛选所有菜肴 dishes = ds.filter(lambda x: '"dish"' in x["food_categories"]) # 筛选所有海鲜类条目 seafood = ds.filter(lambda x: '"seafood"' in x["food_categories"]) # 获取品牌条目 brands = ds.filter(lambda x: x["is_branded_item"]) # 获取非食品条目（可作为训练负样本） non_food = ds.filter(lambda x: not x["is_human_edible"]) # 按规范形式去重（合并"tomatoes"、"Tomatoes"与"tomato"） from collections import defaultdict canonical = defaultdict(int) for row in ds: canonical[row["canonical_form"]] += row["count_total"] top_100 = sorted(canonical.items(), key=lambda x: -x[1])[:100] ## 字段说明 | 字段名 | 数据类型 | 说明 | |---|---|---| | `item` | 字符串 | 原始提取的条目名称（已归一化为小写格式） | | `item_type` | 字符串 | 条目类型："food"或"drink"，即其所属的提取列表 | | `count_re` | 64位整数 | 从`re_caption`（AI生成字幕）中提取该条目的次数 | | `count_org` | 64位整数 | 从`org_caption`（网页替代文本）中提取该条目的次数 | | `count_total` | 64位整数 | `count_re + count_org`的总次数 | | `selection` | 字符串 | 筛选方式："threshold"（出现次数≥100）或"random_sample"（长尾条目的1%随机采样） | | `is_human_edible` | 布尔型 | 人类是否可食用该物品 | | `is_generic_label` | 布尔型 | 是否为模糊描述符（如"红色液体"、"食品"、"饮品"） | | `is_branded_item` | 布尔型 | 是否为品牌名称（如"Coca-Cola"、"Jack Daniel's"） | | `is_raw_ingredient` | 布尔型 | 是否为单一完整未加工原料（如"apple"为是，"apple pie"为否） | | `is_dish` | 布尔型 | 是否为预制/组合菜肴或食谱（如"pad thai"为是，"rice"为否） | | `is_container_or_utensil` | 布尔型 | 是否为容器、器皿或器具（如"wine glass"、"mug"） | | `food_categories` | 字符串（JSON列表格式） | 多标签分类标签 | | `canonical_form` | 字符串 | 规范形式：小写、单数形式且已去除冗余 | ## 分类标签 `food_categories`字段为JSON编码的单标签或多标签列表。例如条目"shrimp pad thai"对应的标签为`["dish", "seafood", "grain"]`。 | 分类标签 | 数量 | 条目占比 | |---|---|---| | `non_food` | 68955 | 23.6% | | `dish` | 56216 | 19.2% | | `drink` | 45314 | 15.5% | | `confectionary` | 32440 | 11.1% | | `baked_goods` | 30878 | 10.5% | | `meat` | 23899 | 8.2% | | `vegetable` | 23113 | 7.9% | | `fruit` | 21802 | 7.4% | | `grain` | 21735 | 7.4% | | `liquor` | 21495 | 7.3% | | `dairy` | 20268 | 6.9% | | `snack` | 16267 | 5.6% | | `other` | 15718 | 5.4% | | `condiments` | 15471 | 5.3% | | `supplement` | 9374 | 3.2% | | `seafood` | 9023 | 3.1% | | `additive` | 8731 | 3.0% | | `herbs_and_spices` | 7106 | 2.4% | | `nuts_and_seeds` | 6097 | 2.1% | | `sweetener` | 5020 | 1.7% | | `frozen_dessert` | 4868 | 1.7% | | `legume` | 4377 | 1.5% | | `spread` | 3522 | 1.2% | | `side_dish` | 2297 | 0.8% | | `eggs` | 2291 | 0.8% | | `oil` | 2026 | 0.7% | | `pet_food` | 1678 | 0.6% | | `cereals` | 1466 | 0.5% | | `fungi` | 1368 | 0.5% | ## 字幕类型说明 条目从两种不同的字幕类型中提取，该信息体现在计数字段中： - **`count_re`**：对应`re_caption`字段，即AI生成的详细字幕（使用LLaVA-1.5-LLaMA3-8B模型），生成通用视觉描述（如"肉类"、"蔬菜"、"酱汁"） - **`count_org`**：对应`org_caption`字段，即原始网页爬取的替代文本，生成具体命名的物品（如"wagyu ribeye"、"pad thai"、"marinara sauce"） 同时使用两种字幕来源可实现互补覆盖。`count_re`值较高的条目在视觉场景中较为常见；`count_org`值较高的条目在网页中被频繁提及。 ## 所用模型 | 处理阶段 | 模型 | 用途 | |---|---|---| | 文本分类 | [mrdbourke/ettin-150m-food-or-drink-classifier](https://huggingface.co/mrdbourke/ettin-150m-food-or-drink-classifier) | 对10亿条字幕进行食品/非食品二分类 | | 食品提取 | [mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2](https://huggingface.co/mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2) | 从字幕中提取条目名称与分类标签 | | 元数据富集 | [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | 为条目添加结构化元数据分类 | ## 数据集局限性 - 条目名称由2.7亿参数模型提取，可能存在提取错误 - 元数据富集标签来自90亿参数模型，并非人工标注 - 长尾条目（出现次数<100）仅按1%比例采样，并非全量覆盖 - `food_categories`字段以JSON字符串格式存储，而非原生列表类型 - 部分规范形式可能无法实现完美去重（如地区拼写差异） - 计数字段反映的是字幕中的提及频率，而非现实世界中的食品流行度 ## 许可证 采用Apache 2.0许可证，与源数据集及所用模型的许可证保持一致。 ## 数据来源 - **源数据集**：[UCSC-VLAA/Recap-DataComp-1B](https://huggingface.co/datasets/UCSC-VLAA/Recap-DataComp-1B) - **过滤后数据集**：[mrdbourke/Recap-DataComp-1B-FoodOrDrink](https://huggingface.co/datasets/mrdbourke/Recap-DataComp-1B-FoodOrDrink) - **食品提取模型**：[mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2](https://huggingface.co/mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2) - **元数据富集模型**：[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)