hanhainebula/BGE-Benchmark-Examples

# BGE Benchmark Examples ## Data Format - `special_examples.jsonl`: each domain (wiki, news) for each language (en, zh) has 3 examples, 12 examples in total. Each example has the following format: ```python { "domain": str, "language": str, "text": str, "characters": List[str], "sampled_characters": List[{ "character": str, "scenarios": List[str], "sampled_scenarios": List[{ "scenario": str, "result": { "prompt1-1": List[{ "query": str, "diversify": { "informal": str, "complicated": str, }, "hard_negative": str, }], "prompt2-1": ..., "prompt3-1": ..., "prompt4-1": ..., "prompt1-2": ..., "prompt2-2": ..., "prompt3-2": ..., "prompt4-2": ..., }, }], }], } ``` - `wiki-news_en-zh_200.jsonl`: each domain (wiki, news) for each language (en, zh) has 50 examples, 200 examples in total. Each example has the following format: ```python {'query': str, 'positive': str, 'hard_negative': str} ``` ## Method LLM: `gpt-4-turbo-preview` For each text → Generate n1 characters → Sample n2 characters ⇒ For each character → Generate n3 scenarios → Sample n4 scenarios ⇒ For each scenario → Generate n5 queries for each prompt (*8 kinds of prompts*) ⇒ For each query → Diversify it in 2 methods & Generate 1 hard negative ⇒ DONE for one example. There are some human-designed characters and scenarios used when generating queries: ```python HUMAN_WRITTEN_CHARACTERS = [ { 'character': 'Professor', 'scenarios': [ 'Setupping questions for an upcoming quiz/examination', 'Preparing for a lecture', ], }, { 'character': 'College student', 'scenarios': [ 'Preparing for an examination', 'Writing a research paper', ], }, { 'character': 'High school student', 'scenarios': [ 'Learning new knowledge', 'Preparing for a presentation', ], } ] ``` For examples in `special_examples.jsonl`, we set n1 = 10 + 3, n2 = 2, n3 = 10 ( + 2), n4 = 2, n5 = 2. For examples in `wiki-news_en-zh_200.jsonl`, we set n1 = 10 + 3, n2 = 1, n3 = 10 ( + 2), n4 = 1, n5 = 3. To generate example in a faster way, we randomly choose one prompt from the 8 kinds of prompts, instead of genrating n5 queries for each prompt (*8 kinds of prompts*).# BGE基准测试示例集 ## 数据格式 ### `special_examples.jsonl` 各语言（英语、中文）的各领域（维基百科、新闻）均包含3条示例，总计12条示例。单条示例的格式如下： python { "domain": str, "language": str, "text": str, "characters": List[str], "sampled_characters": List[{ "character": str, "scenarios": List[str], "sampled_scenarios": List[{ "scenario": str, "result": { "prompt1-1": List[{ "query": str, "diversify": { "informal": str, "complicated": str, }, "hard_negative": str, }], "prompt2-1": ..., "prompt3-1": ..., "prompt4-1": ..., "prompt1-2": ..., "prompt2-2": ..., "prompt3-2": ..., "prompt4-2": ..., }, }], }], } ### `wiki-news_en-zh_200.jsonl` 各语言（英语、中文）的各领域（维基百科、新闻）均包含50条示例，总计200条示例。单条示例的格式如下： python {'query': str, 'positive': str, 'hard_negative': str} ## 构建流程 所用大语言模型：`gpt-4-turbo-preview` 单条示例的构建逻辑如下：针对单条文本，先生成n1个角色（character），再从中采样n2个目标角色；针对每个选中的角色，生成n3个应用场景（scenario），再从中采样n4个目标场景；针对每个选中的场景，为8类提示词（prompt）分别生成n5条查询语句（query）；针对每条查询语句，通过2种方式完成多样化处理，并生成1条难负样本（hard_negative），至此单条示例构建完成。 生成查询语句时，会使用部分人工预设的角色与场景，具体定义如下： python HUMAN_WRITTEN_CHARACTERS = [ { 'character': 'Professor', 'scenarios': [ 'Setupping questions for an upcoming quiz/examination', 'Preparing for a lecture', ], }, { 'character': 'College student', 'scenarios': [ 'Preparing for an examination', 'Writing a research paper', ], }, { 'character': 'High school student', 'scenarios': [ 'Learning new knowledge', 'Preparing for a presentation', ], } ] 针对`special_examples.jsonl`中的示例，参数配置为：n1 = 10 + 3，n2 = 2，n3 = 10 ( + 2)，n4 = 2，n5 = 2。 针对`wiki-news_en-zh_200.jsonl`中的示例，参数配置为：n1 = 10 + 3，n2 = 1，n3 = 10 ( + 2)，n4 = 1，n5 = 3。为提升生成效率，我们不会为8类提示词分别生成n5条查询语句，而是从8类提示词中随机选取一类来生成查询语句。

原始信息汇总

BGE Benchmark Examples 数据集概述

数据格式

special_examples.jsonl

• 每个领域（wiki, news）和每种语言（en, zh）各有3个示例，总共12个示例。
• 每个示例的格式如下： python { "domain": str, "language": str, "text": str, "characters": List[str], "sampled_characters": List[{ "character": str, "scenarios": List[str], "sampled_scenarios": List[{ "scenario": str, "result": { "prompt1-1": List[{ "query": str, "diversify": { "informal": str, "complicated": str, }, "hard_negative": str, }], "prompt2-1": ..., "prompt3-1": ..., "prompt4-1": ..., "prompt1-2": ..., "prompt2-2": ..., "prompt3-2": ..., "prompt4-2": ..., }, }], }], }

wiki-news_en-zh_200.jsonl

• 每个领域（wiki, news）和每种语言（en, zh）各有50个示例，总共200个示例。
• 每个示例的格式如下： python {query: str, positive: str, hard_negative: str}

生成方法

• 使用 gpt-4-turbo-preview 模型。
• 生成过程包括：
  1. 对每个文本生成 n1 个角色。
  2. 从 n1 个角色中采样 n2 个角色。
  3. 对每个角色生成 n3 个场景。
  4. 从 n3 个场景中采样 n4 个场景。
  5. 对每个场景生成 n5 个查询（针对8种提示）。
  6. 对每个查询进行两种方式的多样化处理并生成1个硬负例。

人工设计的角色和场景

python HUMAN_WRITTEN_CHARACTERS = [ { character: Professor, scenarios: [ Setupping questions for an upcoming quiz/examination, Preparing for a lecture, ], }, { character: College student, scenarios: [ Preparing for an examination, Writing a research paper, ], }, { character: High school student, scenarios: [ Learning new knowledge, Preparing for a presentation, ], } ]

参数设置

• 对于 special_examples.jsonl：

  • n1 = 10 + 3
  • n2 = 2
  • n3 = 10 ( + 2)
  • n4 = 2
  • n5 = 2

• 对于 wiki-news_en-zh_200.jsonl：

  • n1 = 10 + 3
  • n2 = 1
  • n3 = 10 ( + 2)
  • n4 = 1
  • n5 = 3
  • 为了更快生成示例，随机选择8种提示中的一种，而不是为每种提示生成 n5 个查询。